AWS Glue catalog

A Glue catalog connects an AWS Glue Data Catalog, so its databases appear in Gravitino as schemas and its Hive, Iceberg, Delta, and Parquet tables can be listed, created, altered, and governed. Changes are written to Glue, where Athena, EMR, Spark, and Trino see them.

Quick start

The example connects the Glue Data Catalog in us-east-1 as the catalog glue_catalog and lists its databases as schemas, then the tables in one of them. Without an access key in the request, the server reaches Glue with the default AWS credential chain, such as the pod's IAM role.

UI

1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter glue_catalog as the Catalog name.

2. Choose the provider. Keep Relational as the Catalog type and choose AWS Glue under Metastores and catalog services in Select provider. Under Connection, enter us-east-1 as the AWS region and s3://{bucket}/warehouse as the Warehouse. Leave Access key ID and Secret access key blank to use the server's default AWS credentials.

3. Create the catalog. Click Test connection, then Create catalog.

4. Browse the catalog. Open Explore and open glue_catalog to list its Glue databases as schemas. Open a schema to list its tables.

REST

1. Create the catalog. warehouse is where tables created through Gravitino go when neither the table nor its database names a location.

   Create a Glue catalog

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "glue_catalog",
       "type": "RELATIONAL",
       "provider": "glue",
       "comment": "AWS Glue Data Catalog",
       "properties": {
         "aws-region": "us-east-1",
         "warehouse": "s3://{bucket}/warehouse"
       }
     }'

   To use a static key instead, add aws-access-key-id and aws-secret-access-key.

2. List the schemas. They are the Glue databases, with names in lowercase.

   List the schemas in the catalog

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

3. List the tables in a database. Hive, Iceberg, Delta, and Parquet tables are all listed, unless table-format-filter narrows them.

   List the tables in a schema

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/glue_catalog/schemas/{database}/tables" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

Requirements

• Network access from the Gravitino server pod to the AWS Glue API, directly or through a VPC endpoint set in aws-glue-endpoint.
• AWS credentials with the IAM permissions below: either an access key in the catalog properties, or the default AWS credential chain of the server pod. On EKS, the default chain picks up an IAM role given to the server's service account through gravitino.serviceAccount.annotations; see Customize the Helm chart.

Schema and table names are case-insensitive, because Glue stores database and table names in lowercase.

Catalog

Capabilities

The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog.

• Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default.
• Supports Hive-format table partitioning, bucketing, and sort orders.
• Does not support views. Glue views (tables with TableType=VIRTUAL_VIEW) are filtered out.

Properties

Besides the common catalog properties, the Glue catalog has the following properties:

Property Name	Description	Default Value	Required	Immutable
aws-region	AWS region for the Glue Data Catalog (e.g. us-east-1).	(none)	Yes	Yes
aws-glue-catalog-id	The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID.	(none)	No	Yes
aws-access-key-id	AWS access key ID for static credential authentication. When omitted, the default credential chain is used.	(none)	No	No
aws-secret-access-key	AWS secret access key paired with aws-access-key-id. When omitted, the default credential chain is used.	(none)	No	No
aws-glue-endpoint	Glue endpoint to call instead of the regional default, such as a VPC endpoint URL.	(none)	No	No
warehouse	Base storage path used as the warehouse when no explicit location is specified at table creation time (e.g. s3://my-bucket/warehouse). Table location is derived as warehouse/database/table.	(none)	Yes	No
default-table-format	Default format for tables created via Gravitino's createTable() API. Accepted values: iceberg, hive.	hive	No	No
table-format-filter	Comma-separated list of table formats exposed by listTables() and loadTable(). Accepted values: all, hive, iceberg, delta, parquet. Use to restrict visible table types.	all	No	No

An access key in aws-access-key-id and aws-secret-access-key takes precedence over the default credential chain of environment variables, the service account's IAM role, and the instance profile.

Operations

Refer to Catalogs and schemas for more details.

aws-access-key-id and aws-secret-access-key are hidden when the catalog is loaded; engines get storage access through credential vending.

Schema

Capabilities

The Glue catalog supports creating, updating, and deleting databases in the AWS Glue Data Catalog.

Properties

A schema has no properties of its own beyond comment. Any property you set is stored on the Glue database.

Operations

See Catalogs and schemas.

Table

Capabilities

• The Glue catalog supports creating, updating, and deleting tables in the AWS Glue Data Catalog.
• All entries in the Glue Table.parameters() pass through Gravitino intact, so downstream tools can correctly identify the table format.
• Does not support column default value.
• Does not support NOT NULL constraints on columns.
• Does not support table indexes.

Partitioning

The Glue catalog supports partitioned tables. Create partitioned tables in the Glue catalog by specifying the partitioning attribute.

The supported partitioning strategies depend on the table format:

• Hive-format tables: Only Identity partitioning is supported, because the native Glue partition model is Hive-style key=value.
• Iceberg-format tables: All Iceberg partition transforms are supported: identity, year, month, day, hour, bucket, and truncate.

The fieldName specified in the partitioning attribute must be the name of a column defined in the table.

Sort orders and distributions

The Glue catalog supports bucketed sorted tables. Create bucketed sorted tables by setting the distribution and sortOrders attributes. Although Gravitino supports several distribution strategies, AWS Glue inherently only supports a single distribution strategy (clustered by column). Therefore, the Glue catalog only supports Hash distribution.

The fieldName specified in the distribution and sortOrders attribute must be the name of a column defined in the table.

Column types

The Glue catalog supports all data types defined in the Hive Language Manual. The following table lists the data types mapped from the Glue catalog to Gravitino.

Glue Data Type	Gravitino Data Type
boolean	boolean
tinyint	byte
smallint	short
int / integer	integer
bigint	long
float	float
double	double
decimal	decimal
string	string
char	char
varchar	varchar
timestamp	timestamp
date	date
interval_year_month	interval_year
interval_day_time	interval_day
binary	binary
array	list
map	map
struct	struct
uniontype	union

Data types not listed above map to Gravitino External Type, which represents an unresolvable data type from the Glue catalog.

Properties

Besides the properties below, any property you set is stored as a Glue table parameter. Reserved properties cannot be set by you; immutable ones cannot change after the table is created.

Property Name	Description	Default Value	Required	Reserved	Immutable
location	The location for table storage, such as s3://bucket/prefix/test_table. Derived from warehouse/database/table when not specified.	(derived from warehouse)	No	No	No
format	The table file format (parquet, orc, textfile, etc.). When set, input-format, output-format, and serde-lib are derived automatically. Used primarily when creating Hive-format tables via Trino.	(none)	No	No	Yes
input-format	The input format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.	org.apache.hadoop.mapred.TextInputFormat	No	No	Yes
output-format	The output format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.	org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat	No	No	Yes
serde-lib	The serde library class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcSerde.	org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe	No	No	Yes
serde-name	The name of the serde.	(none)	No	No	No
serde.parameter.	The prefix of the serde parameter, such as "serde.parameter.orc.create.index" = "true", indicating ORC serde lib to create row indexes.	(none)	No	No	No
table-format	Table format stored in Glue Table.parameters(). Use ICEBERG to create an Iceberg table. Common values: ICEBERG, HIVE.	(none)	No	No	No
metadata_location	Iceberg table metadata file location stored in Glue Table.parameters(). When set during createTable(), registers an existing Iceberg table rather than creating a new one.	(none)	No	No	No
comment	Used to store a table comment.	(none)	No	Yes	No

Every Glue table parameter passes through Gravitino unchanged, so format markers such as table_type=ICEBERG, metadata_location, and spark.sql.sources.provider=delta reach engines intact.

Operations

Refer to Tables and views for more details.

Alter operations

Gravitino defines a unified set of metadata operation interfaces. The following table maps Glue alter operations to Gravitino table update requests.

Alter table

Glue Alter Operation	Gravitino Table Update Request
Alter Table Properties	Set a table property
Alter Table Comment	Update comment
Remove Properties	Remove a table property

Hive-format table rename is not supported. AWS Glue does not provide a native rename API for tables; renaming would require recreating the table. Iceberg-format table rename is supported.

Alter column

Glue Alter Operation	Gravitino Table Update Request
Change Column Name	Rename a column
Change Column Type	Update the type of a column
Change Column Position	Update the position of a column
Change Column Comment	Update the column comment

Alter partition

The Glue catalog supports partition operations via SupportsPartitions for Hive-format identity-partitioned tables:

• listPartitions() / listPartitionNames()
• getPartition(partitionName)
• addPartition(partition)
• dropPartition(partitionName)

Only IdentityPartition is supported because the Glue partition model is Hive-style key=value.

Iceberg tables

The Glue catalog supports creating and managing Iceberg-format tables through the Apache Iceberg SDK's GlueCatalog. When an Iceberg table is created, Gravitino writes the metadata.json file to S3 and registers the table in Glue with the correct metadata_location parameter, making it usable by Trino (Lakehouse connector), Spark, and other Iceberg-native query engines.

Create

Set table-format=ICEBERG in the table properties, or configure default-table-format=iceberg on the catalog to make all tables Iceberg by default.

The warehouse catalog property must be configured. The table location is derived as warehouse/database/table when no explicit location is specified.

Register an existing table

To register an Iceberg table that already exists in S3, set metadata_location to the path of the existing metadata.json file during createTable(). In this mode, Gravitino registers the table in Glue without creating new metadata.

Column types

Iceberg tables use the Iceberg type system, which differs from Hive types. The following table lists the Gravitino types supported for Iceberg tables and how they map to Iceberg types:

Gravitino Data Type	Iceberg Data Type	Notes
boolean	boolean
byte	int	Widened to 32-bit integer
short	int	Widened to 32-bit integer
integer	int
long	long
float	float
double	double
decimal(p, s)	decimal(p, s)
string	string
varchar	string	Iceberg has no variable-length char types
char	string	Iceberg has no variable-length char types
date	date
time(6)	time	Only microsecond precision (6) is supported
timestamp(6)	timestamp	Only microsecond precision (6) is supported
timestamptz(6)	timestamptz	Only microsecond precision (6) is supported
binary	binary
fixed(n)	binary	Mapped to variable-length binary
uuid	uuid
list	list
map	map
struct	struct

Alter operations

For Iceberg tables, the following alter operations are supported:

Operation	Gravitino Table Update Request
Add column	Add a column
Delete column	Delete a column
Rename column	Rename a column
Update column type	Update the type of a column
Update column comment	Update the column comment
Update column nullability	Update column nullability
Set table property	Set a table property
Remove table property	Remove a table property

warning

Schema changes and property changes are committed in two separate Iceberg transactions. If the schema commit succeeds but the property commit fails, the table is left in a partially altered state.

Nested column operations (add, delete, rename, type update) are not supported for Iceberg tables via this catalog.

Security

AWS IAM permissions

The IAM policy attached to the credential used by the Glue catalog must cover both Glue metadata access and S3 data access:

Glue catalog IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueMetadataAccess",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog",
        "glue:GetDatabase", "glue:GetDatabases",
        "glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
        "glue:GetTable", "glue:GetTables",
        "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
        "glue:GetPartition", "glue:GetPartitions",
        "glue:CreatePartition", "glue:DeletePartition"
      ],
      "Resource": [
        "arn:aws:glue:{region}:{account_id}:catalog",
        "arn:aws:glue:{region}:{account_id}:database/*",
        "arn:aws:glue:{region}:{account_id}:table/*/*"
      ]
    },
    {
      "Sid": "S3DataAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::{warehouse_bucket}",
        "arn:aws:s3:::{warehouse_bucket}/*"
      ]
    }
  ]
}

Related

• Catalogs and schemas
• Tables and views
• Partitions
• Credential vending