AWS Glue catalog
A Glue catalog connects an AWS Glue Data Catalog, so its databases appear in Gravitino as schemas and its Hive, Iceberg, Delta, and Parquet tables can be listed, created, altered, and governed. Changes are written to Glue, where Athena, EMR, Spark, and Trino see them.
Quick start
The example connects the Glue Data Catalog in us-east-1 as the catalog glue_catalog and lists its
databases as schemas, then the tables in one of them. Without an access key in the request, the
server reaches Glue with the default AWS credential chain, such as the pod's IAM role.
- UI
- REST
-
Start the catalog. Open Connect → Catalogs and click + Catalog. Enter
glue_catalogas the Catalog name. -
Choose the provider. Keep Relational as the Catalog type and choose AWS Glue under Metastores and catalog services in Select provider. Under Connection, enter
us-east-1as the AWS region ands3://{bucket}/warehouseas the Warehouse. Leave Access key ID and Secret access key blank to use the server's default AWS credentials. -
Create the catalog. Click Test connection, then Create catalog.
-
Browse the catalog. Open Explore and open
glue_catalogto list its Glue databases as schemas. Open a schema to list its tables.
-
Create the catalog.
warehouseis where tables created through Gravitino go when neither the table nor its database names a location.Create a Glue catalogcurl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"name": "glue_catalog",
"type": "RELATIONAL",
"provider": "glue",
"comment": "AWS Glue Data Catalog",
"properties": {
"aws-region": "us-east-1",
"warehouse": "s3://{bucket}/warehouse"
}
}'To use a static key instead, add
aws-access-key-idandaws-secret-access-key. -
List the schemas. They are the Glue databases, with names in lowercase.
List the schemas in the catalogcurl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Authorization: Bearer $TOKEN" -
List the tables in a database. Hive, Iceberg, Delta, and Parquet tables are all listed, unless
table-format-filternarrows them.List the tables in a schemacurl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas/{database}/tables" \
-H "Accept: application/vnd.gravitino.v1+json" \
-H "Authorization: Bearer $TOKEN"
Requirements
- Network access from the Gravitino server pod to the AWS Glue API, directly or through a VPC
endpoint set in
aws-glue-endpoint. - AWS credentials with the IAM permissions below: either an access key in the
catalog properties, or the default AWS credential chain of the server pod. On EKS, the default
chain picks up an IAM role given to the server's service account through
gravitino.serviceAccount.annotations; see Customize the Helm chart.
Schema and table names are case-insensitive, because Glue stores database and table names in lowercase.
Catalog
Capabilities
The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog.
- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default.
- Supports Hive-format table partitioning, bucketing, and sort orders.
- Does not support views. Glue views (tables with
TableType=VIRTUAL_VIEW) are filtered out.
Properties
Besides the common catalog properties, the Glue catalog has the following properties:
| Property Name | Description | Default Value | Required | Immutable |
|---|---|---|---|---|
aws-region | AWS region for the Glue Data Catalog (e.g. us-east-1). | (none) | Yes | Yes |
aws-glue-catalog-id | The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID. | (none) | No | Yes |
aws-access-key-id | AWS access key ID for static credential authentication. When omitted, the default credential chain is used. | (none) | No | No |
aws-secret-access-key | AWS secret access key paired with aws-access-key-id. When omitted, the default credential chain is used. | (none) | No | No |
aws-glue-endpoint | Glue endpoint to call instead of the regional default, such as a VPC endpoint URL. | (none) | No | No |
warehouse | Base storage path used as the warehouse when no explicit location is specified at table creation time (e.g. s3://my-bucket/warehouse). Table location is derived as warehouse/database/table. | (none) | Yes | No |
default-table-format | Default format for tables created via Gravitino's createTable() API. Accepted values: iceberg, hive. | hive | No | No |
table-format-filter | Comma-separated list of table formats exposed by listTables() and loadTable(). Accepted values: all, hive, iceberg, delta, parquet. Use to restrict visible table types. | all | No | No |
An access key in aws-access-key-id and aws-secret-access-key takes precedence over the default credential chain of environment variables, the service account's IAM role, and the instance profile.
Operations
Refer to Catalogs and schemas for more details.
aws-access-key-id and aws-secret-access-key are hidden when the catalog is loaded; engines get storage access through credential vending.
Schema
Capabilities
The Glue catalog supports creating, updating, and deleting databases in the AWS Glue Data Catalog.
Properties
A schema has no properties of its own beyond comment. Any property you set is stored on the Glue database.
Operations
See Catalogs and schemas.
Table
Capabilities
- The Glue catalog supports creating, updating, and deleting tables in the AWS Glue Data Catalog.
- All entries in the Glue
Table.parameters()pass through Gravitino intact, so downstream tools can correctly identify the table format. - Does not support column default value.
- Does not support NOT NULL constraints on columns.
- Does not support table indexes.
Partitioning
The Glue catalog supports partitioned tables. Create partitioned tables in the Glue catalog by specifying the partitioning attribute.
The supported partitioning strategies depend on the table format:
- Hive-format tables: Only
Identitypartitioning is supported, because the native Glue partition model is Hive-style key=value. - Iceberg-format tables: All Iceberg partition transforms are supported:
identity,year,month,day,hour,bucket, andtruncate.
The fieldName specified in the partitioning attribute must be the name of a column defined in the table.
Sort orders and distributions
The Glue catalog supports bucketed sorted tables. Create bucketed sorted tables by setting the distribution and sortOrders attributes.
Although Gravitino supports several distribution strategies, AWS Glue inherently only supports a single distribution strategy (clustered by column). Therefore, the Glue catalog only supports Hash distribution.
The fieldName specified in the distribution and sortOrders attribute must be the name of a column defined in the table.
Column types
The Glue catalog supports all data types defined in the Hive Language Manual. The following table lists the data types mapped from the Glue catalog to Gravitino.
| Glue Data Type | Gravitino Data Type |
|---|---|
boolean | boolean |
tinyint | byte |
smallint | short |
int / integer | integer |
bigint | long |
float | float |
double | double |
decimal | decimal |
string | string |
char | char |
varchar | varchar |
timestamp | timestamp |
date | date |
interval_year_month | interval_year |
interval_day_time | interval_day |
binary | binary |
array | list |
map | map |
struct | struct |
uniontype | union |
Data types not listed above map to Gravitino External Type, which represents an unresolvable data type from the Glue catalog.
Properties
Besides the properties below, any property you set is stored as a Glue table parameter. Reserved properties cannot be set by you; immutable ones cannot change after the table is created.
| Property Name | Description | Default Value | Required | Reserved | Immutable |
|---|---|---|---|---|---|
location | The location for table storage, such as s3://bucket/prefix/test_table. Derived from warehouse/database/table when not specified. | (derived from warehouse) | No | No | No |
format | The table file format (parquet, orc, textfile, etc.). When set, input-format, output-format, and serde-lib are derived automatically. Used primarily when creating Hive-format tables via Trino. | (none) | No | No | Yes |
input-format | The input format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcInputFormat. | org.apache.hadoop.mapred.TextInputFormat | No | No | Yes |
output-format | The output format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat. | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | No | No | Yes |
serde-lib | The serde library class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcSerde. | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | No | No | Yes |
serde-name | The name of the serde. | (none) | No | No | No |
serde.parameter. | The prefix of the serde parameter, such as "serde.parameter.orc.create.index" = "true", indicating ORC serde lib to create row indexes. | (none) | No | No | No |
table-format | Table format stored in Glue Table.parameters(). Use ICEBERG to create an Iceberg table. Common values: ICEBERG, HIVE. | (none) | No | No | No |
metadata_location | Iceberg table metadata file location stored in Glue Table.parameters(). When set during createTable(), registers an existing Iceberg table rather than creating a new one. | (none) | No | No | No |
comment | Used to store a table comment. | (none) | No | Yes | No |
Every Glue table parameter passes through Gravitino unchanged, so format markers such as table_type=ICEBERG, metadata_location, and spark.sql.sources.provider=delta reach engines intact.
Operations
Refer to Tables and views for more details.
Alter operations
Gravitino defines a unified set of metadata operation interfaces. The following table maps Glue alter operations to Gravitino table update requests.
Alter table
| Glue Alter Operation | Gravitino Table Update Request |
|---|---|
Alter Table Properties | Set a table property |
Alter Table Comment | Update comment |
Remove Properties | Remove a table property |
Hive-format table rename is not supported. AWS Glue does not provide a native rename API for tables; renaming would require recreating the table. Iceberg-format table rename is supported.
Alter column
| Glue Alter Operation | Gravitino Table Update Request |
|---|---|
Change Column Name | Rename a column |
Change Column Type | Update the type of a column |
Change Column Position | Update the position of a column |
Change Column Comment | Update the column comment |
Alter partition
The Glue catalog supports partition operations via SupportsPartitions for Hive-format identity-partitioned tables:
listPartitions()/listPartitionNames()getPartition(partitionName)addPartition(partition)dropPartition(partitionName)
Only IdentityPartition is supported because the Glue partition model is Hive-style key=value.
Iceberg tables
The Glue catalog supports creating and managing Iceberg-format tables through the Apache Iceberg SDK's GlueCatalog. When an Iceberg table is created, Gravitino writes the metadata.json file to S3 and registers the table in Glue with the correct metadata_location parameter, making it usable by Trino (Lakehouse connector), Spark, and other Iceberg-native query engines.
Create
Set table-format=ICEBERG in the table properties, or configure default-table-format=iceberg on the catalog to make all tables Iceberg by default.
The warehouse catalog property must be configured. The table location is derived as warehouse/database/table when no explicit location is specified.
Register an existing table
To register an Iceberg table that already exists in S3, set metadata_location to the path of the existing metadata.json file during createTable(). In this mode, Gravitino registers the table in Glue without creating new metadata.
Column types
Iceberg tables use the Iceberg type system, which differs from Hive types. The following table lists the Gravitino types supported for Iceberg tables and how they map to Iceberg types:
| Gravitino Data Type | Iceberg Data Type | Notes |
|---|---|---|
boolean | boolean | |
byte | int | Widened to 32-bit integer |
short | int | Widened to 32-bit integer |
integer | int | |
long | long | |
float | float | |
double | double | |
decimal(p, s) | decimal(p, s) | |
string | string | |
varchar | string | Iceberg has no variable-length char types |
char | string | Iceberg has no variable-length char types |
date | date | |
time(6) | time | Only microsecond precision (6) is supported |
timestamp(6) | timestamp | Only microsecond precision (6) is supported |
timestamptz(6) | timestamptz | Only microsecond precision (6) is supported |
binary | binary | |
fixed(n) | binary | Mapped to variable-length binary |
uuid | uuid | |
list | list | |
map | map | |
struct | struct |
Alter operations
For Iceberg tables, the following alter operations are supported:
| Operation | Gravitino Table Update Request |
|---|---|
| Add column | Add a column |
| Delete column | Delete a column |
| Rename column | Rename a column |
| Update column type | Update the type of a column |
| Update column comment | Update the column comment |
| Update column nullability | Update column nullability |
| Set table property | Set a table property |
| Remove table property | Remove a table property |
Schema changes and property changes are committed in two separate Iceberg transactions. If the schema commit succeeds but the property commit fails, the table is left in a partially altered state.
Nested column operations (add, delete, rename, type update) are not supported for Iceberg tables via this catalog.
Security
AWS IAM permissions
The IAM policy attached to the credential used by the Glue catalog must cover both Glue metadata access and S3 data access:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GlueMetadataAccess",
"Effect": "Allow",
"Action": [
"glue:GetCatalog",
"glue:GetDatabase", "glue:GetDatabases",
"glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
"glue:GetTable", "glue:GetTables",
"glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
"glue:GetPartition", "glue:GetPartitions",
"glue:CreatePartition", "glue:DeletePartition"
],
"Resource": [
"arn:aws:glue:{region}:{account_id}:catalog",
"arn:aws:glue:{region}:{account_id}:database/*",
"arn:aws:glue:{region}:{account_id}:table/*/*"
]
},
{
"Sid": "S3DataAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::{warehouse_bucket}",
"arn:aws:s3:::{warehouse_bucket}/*"
]
}
]
}