Skip to main content

AWS Glue catalog

A Glue catalog connects an AWS Glue Data Catalog, so its databases appear in Gravitino as schemas and its Hive, Iceberg, Delta, and Parquet tables can be listed, created, altered, and governed. Changes are written to Glue, where Athena, EMR, Spark, and Trino see them.

Quick start​

The example connects the Glue Data Catalog in us-east-1 as the catalog glue_catalog and lists its databases as schemas, then the tables in one of them. Without an access key in the request, the server reaches Glue with the default AWS credential chain, such as the pod's IAM role.

  1. Start the catalog. Open Connect → Catalogs and click + Catalog. Enter glue_catalog as the Catalog name.

  2. Choose the provider. Keep Relational as the Catalog type and choose AWS Glue under Metastores and catalog services in Select provider. Under Connection, enter us-east-1 as the AWS region and s3://{bucket}/warehouse as the Warehouse. Leave Access key ID and Secret access key blank to use the server's default AWS credentials.

  3. Create the catalog. Click Test connection, then Create catalog.

  4. Browse the catalog. Open Explore and open glue_catalog to list its Glue databases as schemas. Open a schema to list its tables.

Requirements​

  • Network access from the Gravitino server pod to the AWS Glue API, directly or through a VPC endpoint set in aws-glue-endpoint.
  • AWS credentials with the IAM permissions below: either an access key in the catalog properties, or the default AWS credential chain of the server pod. On EKS, the default chain picks up an IAM role given to the server's service account through gravitino.serviceAccount.annotations; see Customize the Helm chart.

Schema and table names are case-insensitive, because Glue stores database and table names in lowercase. Column names of Hive-format tables are stored in lowercase too. Iceberg tables keep the case of their column names in the Iceberg metadata, but the copy of the columns in Glue is lowercase, so some engines cannot read mixed-case Iceberg columns.

Manage each Glue Data Catalog (one AWS account, region and aws-glue-catalog-id) from only one Gravitino catalog, in the same way that two Hive catalogs must not share one Hive Metastore. Two Gravitino catalogs on the same Glue Data Catalog are not supported.

Catalog​

Capabilities​

The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog.

  • Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default.
  • Supports Hive-format table partitioning, bucketing, and sort orders.
  • Does not support views. Glue views (tables with TableType=VIRTUAL_VIEW) are filtered out.

Properties​

Besides the common catalog properties, the Glue catalog has the following properties:

Property NameDescriptionDefault ValueRequiredImmutable
aws-regionAWS region for the Glue Data Catalog, as a known AWS region identifier in lower case (e.g. us-east-1).(none)YesYes
aws-glue-catalog-idThe 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID.(none)NoYes
aws-access-key-idAWS access key ID for static credential authentication. When omitted, the default credential chain is used. Temporary keys (prefix ASIA) are refused, because they need a session token.(none)NoNo
aws-secret-access-keyAWS secret access key paired with aws-access-key-id. When omitted, the default credential chain is used.(none)NoNo
aws-glue-endpointGlue endpoint to call instead of the regional default, such as a VPC endpoint URL. Must be an http or https URL.(none)NoNo
aws-glue-api-call-timeout-msUpper bound in milliseconds for one Glue API call, including its retries. Applies to the Glue calls of Iceberg tables too, but not to S3 reads and writes.30000NoNo
aws-glue-api-call-attempt-timeout-msUpper bound in milliseconds for a single attempt of a Glue API call. Also the connection and socket timeout of the S3 client used for Iceberg tables.10000NoNo
aws-glue-max-attemptsMaximum number of attempts of a Glue API call, including the first. Throttling and transient failures are retried. Applies to the Glue calls of Iceberg tables too, but not to S3 I/O.4NoNo
warehouseBase storage path used when a table is created without a location and its Glue database declares no LocationUri (e.g. s3://my-bucket/warehouse). The table location is then warehouse/database/table. Must be a URI with a scheme; Iceberg tables need s3://, s3a:// or s3n://.(none)YesNo
default-table-formatDefault format for tables created via Gravitino's createTable() API. Accepted values: iceberg, hive.hiveNoNo
table-format-filterComma-separated list of table formats the catalog exposes. Accepted values: all, hive, iceberg, delta, parquet. A table is iceberg when it has table_type=ICEBERG and hive otherwise, including Delta tables registered by other engines, so delta and parquet match no table. Tables of other formats are absent from list, load, alter and drop, and cannot be created.allNoNo

An access key in aws-access-key-id and aws-secret-access-key takes precedence over the default credential chain of environment variables, the service account's IAM role, and the instance profile.

Operations​

Refer to Catalogs and schemas for more details.

aws-access-key-id and aws-secret-access-key are hidden when the catalog is loaded; engines get storage access through credential vending.

Schema​

Capabilities​

The Glue catalog supports creating, updating, and deleting databases in the AWS Glue Data Catalog.

Properties​

A schema has no properties of its own beyond comment. Any property you set is stored on the Glue database.

Operations​

See Catalogs and schemas.

Table​

Capabilities​

  • The Glue catalog supports creating, updating, and deleting tables in the AWS Glue Data Catalog.
  • All entries in the Glue Table.parameters() pass through Gravitino intact, so downstream tools can correctly identify the table format.
  • Does not support column default value.
  • Does not support NOT NULL constraints on columns of Hive-format tables, including on nested fields (list elements, map values and struct fields). Iceberg tables support them.
  • char and varchar columns of Hive-format tables follow the Hive length limits: char takes 1 to 255 and varchar 1 to 65535. Other lengths are refused.
  • Does not support table indexes.

Partitioning​

The Glue catalog supports partitioned tables. Create partitioned tables in the Glue catalog by specifying the partitioning attribute.

The supported partitioning strategies depend on the table format:

  • Hive-format tables: Only Identity partitioning is supported, because the native Glue partition model is Hive-style key=value.
  • Iceberg-format tables: All Iceberg partition transforms are supported: identity, year, month, day, hour, bucket, and truncate.

The fieldName specified in the partitioning attribute must be the name of a column defined in the table. For Hive-format tables, the partition columns must be the last columns of the table, in partitioning order, and must have a primitive type.

Sort orders and distributions​

The Glue catalog supports bucketed sorted tables. Create bucketed sorted tables by setting the distribution and sortOrders attributes. Although Gravitino supports several distribution strategies, AWS Glue inherently only supports a single distribution strategy (clustered by column). Therefore, the Glue catalog only supports Hash distribution.

The fieldName specified in the distribution and sortOrders attribute must be the name of a column defined in the table. Only plain column references are supported, and the number of buckets must be at least 1. A sort order must use the default null ordering of its direction (nulls_first ascending, nulls_last descending), because Glue stores only the direction.

Column types​

The Glue catalog supports all data types defined in the Hive Language Manual. The following table lists the data types mapped from the Glue catalog to Gravitino.

Glue Data TypeGravitino Data Type
booleanboolean
tinyintbyte
smallintshort
int / integerinteger
bigintlong
floatfloat
doubledouble
double precisiondouble
decimaldecimal
decimal (no args)decimal(10,0)
stringstring
charchar
varcharvarchar
timestamptimestamp
datedate
interval_year_monthinterval_year
interval_day_timeinterval_day
binarybinary
arraylist
mapmap
structstruct
uniontypeunion

Data types not listed above map to Gravitino External Type, which represents an unresolvable data type from the Glue catalog.

Unsigned integer types are refused, because Glue and Hive have signed integers only. Struct field names must not contain :, ,, < or >, because Glue type strings cannot quote them.

Properties​

Besides the properties below, any property you set is stored as a Glue table parameter. Reserved properties cannot be set by you; immutable ones cannot change after the table is created.

Property NameDescriptionDefault ValueRequiredReservedImmutable
locationThe location for table storage, such as s3://bucket/prefix/test_table. When not set, it is the Glue database's LocationUri followed by the table name, or warehouse/database/table when the database declares no location.(derived from database location or warehouse)NoNoNo
formatThe table file format (parquet, orc, textfile, etc.). When set, input-format, output-format, and serde-lib are derived from it, and any of them also set must match. Without format, set all three classes or none; with none, the table is a text table.(none)NoNoYes
input-formatThe input format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.org.apache.hadoop.mapred.TextInputFormatNoNoYes
output-formatThe output format class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatNoNoYes
serde-libThe serde library class for the table, such as org.apache.hadoop.hive.ql.io.orc.OrcSerde.org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDeNoNoYes
serde-nameThe name of the serde.(none)NoNoNo
serde.parameter.The prefix of the serde parameter, such as "serde.parameter.orc.create.index" = "true", indicating ORC serde lib to create row indexes.(none)NoNoNo
table-formatFormat the table is created in: HIVE or ICEBERG. It is not stored; a loaded table reports ICEBERG when it has table_type=ICEBERG, and HIVE otherwise.(catalog default-table-format)NoNoYes
metadata_locationIceberg table metadata file location stored in Glue Table.parameters(). When set during createTable(), registers an existing Iceberg table rather than creating a new one; see Register an existing table.(none)NoNoNo
commentUsed to store a table comment.(none)NoYesNo

Every Glue table parameter passes through Gravitino unchanged, so format markers such as table_type=ICEBERG, metadata_location, and spark.sql.sources.provider=delta reach engines intact.

Operations​

Refer to Tables and views for more details.

Alter operations​

Gravitino defines a unified set of metadata operation interfaces. The following table maps Glue alter operations to Gravitino table update requests.

Alter table​
Glue Alter OperationGravitino Table Update Request
Alter Table PropertiesSet a table property
Alter Table CommentUpdate comment
Remove PropertiesRemove a table property

Hive-format table rename is not supported. AWS Glue does not provide a native rename API for tables; renaming would require recreating the table. Iceberg-format table rename is supported.

Alter column​
Glue Alter OperationGravitino Table Update Request
Change Column NameRename a column
Change Column TypeUpdate the type of a column
Change Column PositionUpdate the position of a column
Change Column CommentUpdate the column comment

For Hive-format tables, changes to nested struct fields are not supported; change the whole column instead. Partition columns accept only comment changes. Making a column nullable is accepted and changes nothing, since every column is nullable already, and making a column NOT NULL is refused. Bucket and sort columns can be renamed but not deleted.

Alter partition​

The Glue catalog supports partition operations via SupportsPartitions for Hive-format identity-partitioned tables:

  • listPartitions() / listPartitionNames()
  • getPartition(partitionName)
  • addPartition(partition)
  • dropPartition(partitionName)

Only IdentityPartition is supported because the Glue partition model is Hive-style key=value.

Partition names escape special characters in values as Hive does, for example / becomes %2F. A new partition takes its columns, formats and SerDe from the table, and its location is the table location followed by the partition name.

Iceberg tables​

The Glue catalog supports creating and managing Iceberg-format tables through the Apache Iceberg SDK's GlueCatalog. When an Iceberg table is created, Gravitino writes the metadata.json file to S3 and registers the table in Glue with the correct metadata_location parameter, making it usable by Trino (Lakehouse connector), Spark, and other Iceberg-native query engines.

Create​

Set table-format=ICEBERG in the table properties, or configure default-table-format=iceberg on the catalog to make all tables Iceberg by default.

The warehouse catalog property must be configured. The table location is derived as warehouse/database/table when no explicit location is specified.

Register an existing table​

To register an Iceberg table that already exists in S3, set metadata_location to the path of the existing metadata.json file during createTable(). Gravitino reads the file before it creates the Glue table and refuses the request if the file is missing or is not Iceberg table metadata. No new metadata file is written.

The columns, location, partitioning and sort order of a registered table come from the metadata file. Send "columns": [] to take the columns from the file; columns you do send must match it by name. Do not set location, partitioning or sort order, and do not give a property of the metadata file a different value.

A registered table shares its metadata and data files with the table it came from. Gravitino marks it with the Glue parameter gravitino.iceberg.registered=true and afterwards changes only its Glue parameters: property changes are stored in Glue, and column, partition and other schema changes are refused. Properties of the metadata file cannot be set or removed, because Iceberg engines read them from that file. A registered table cannot be purged.

A table registered by Gravitino before 1.3.2, or by a registration that stopped before the parameter was written, has no marker. Gravitino then treats it as the owner of its metadata, so an alter writes new metadata files and forks the history it shares with the source table. Gravitino cannot detect such tables or set the parameter. To protect one, add gravitino.iceberg.registered=true to the table directly in AWS Glue.

Column types​

Iceberg tables use the Iceberg type system, which differs from Hive types. The following table lists the Gravitino types supported for Iceberg tables and how they map to Iceberg types:

Gravitino Data TypeIceberg Data TypeNotes
booleanboolean
integerint
longlong
floatfloat
doubledouble
decimal(p, s)decimal(p, s)
stringstring
varcharstringIceberg has no variable-length char types
charstringIceberg has no variable-length char types
datedate
time(6)timeOnly microsecond precision (6) is supported
timestamp(6)timestamp
timestamptz(6)timestamptz
timestamp(9)timestamp_nsNeeds format version 3
timestamptz(9)timestamptz_nsNeeds format version 3
binarybinary
fixed(n)fixed(n)
uuiduuid
nullunknownNeeds format version 3
listlist
mapmap
structstruct

byte, short and unsigned types are refused for Iceberg tables; use integer or long.

Alter operations​

For Iceberg tables, the following alter operations are supported:

OperationGravitino Table Update Request
Add columnAdd a column
Delete columnDelete a column
Rename columnRename a column
Update column typeUpdate the type of a column
Update column positionUpdate the position of a column
Update column commentUpdate the column comment
Update column nullabilityUpdate column nullability
Set table propertySet a table property
Remove table propertyRemove a table property

Iceberg allows a column to change only from required to optional. Making an optional column required, or adding a required column, is refused as an incompatible schema change.

The schema and property changes of one request are committed in one Iceberg transaction, so they are applied together or not at all. A request that reports an error may still have been applied, for example when the commit state is unknown or the reload after the commit fails, so check the table before retrying. Property changes of a registered table are stored in Glue, outside any Iceberg transaction.

Nested column operations, auto-increment and column default values are not supported for Iceberg tables.

Drop and purge​

Dropping an Iceberg table removes only its Glue entry; the metadata and data files stay in S3. Purging an Iceberg table also deletes the data and metadata files its metadata references. Hive-format tables and registered Iceberg tables cannot be purged.

Partitions​

Iceberg tables keep their partitions in Iceberg metadata, not in Glue, so partition operations are not supported for them. The table itself reports its partitioning. Fields with the void transform, which Iceberg v1 tables keep for dropped partition fields, are left out.

Security​

AWS IAM permissions​

The IAM policy attached to the credential used by the Glue catalog must cover both Glue metadata access and S3 data access:

Glue catalog IAM policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GlueMetadataAccess",
"Effect": "Allow",
"Action": [
"glue:GetCatalog",
"glue:GetDatabase", "glue:GetDatabases",
"glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
"glue:GetTable", "glue:GetTables",
"glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
"glue:GetPartition", "glue:GetPartitions",
"glue:CreatePartition", "glue:DeletePartition"
],
"Resource": [
"arn:aws:glue:{region}:{account_id}:catalog",
"arn:aws:glue:{region}:{account_id}:database/*",
"arn:aws:glue:{region}:{account_id}:table/*/*"
]
},
{
"Sid": "S3DataAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::{warehouse_bucket}",
"arn:aws:s3:::{warehouse_bucket}/*"
]
}
]
}