Delta Lake tables

A generic lakehouse catalog can register existing Delta Lake tables: tables whose format property is delta. Gravitino records the columns, location, and properties you give it and never reads or writes the Delta data or its transaction log. Every Delta table is external, so dropping it from Gravitino leaves the data where it is.

Quick start

The example registers an existing Delta table at s3://{bucket}/delta/orders as orders in the schema sales of the generic lakehouse catalog lakehouse_catalog, set up as in the generic lakehouse catalog Quick start.

1. Register the table. format, external, and location are all required. Gravitino stores the columns as given and does not read them from the Delta log, so list them as the table defines them.

   Register an external Delta table

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "orders",
       "comment": "Orders Delta table",
       "columns": [
         {"name": "order_id", "type": "long", "nullable": false},
         {"name": "total_amount", "type": "decimal(10,2)", "nullable": true}
       ],
       "properties": {
         "format": "delta",
         "external": "true",
         "location": "s3://{bucket}/delta/orders"
       }
     }'

2. Load the table.

   Load a Delta table

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables/orders" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

Dropping the table removes only the registration; the Delta data stays where it is.

Supported operations

Operation	Supported	Notes
Register	Yes	Through create table, with external set to true.
List	Yes
Load	Yes
Alter	No	Change the table with Delta Lake tools, then drop and register it again if its columns changed.
Drop	Yes	Removes the registration only.
Purge	No	Delete the data with Delta Lake tools or on storage.

Partitions can be recorded with identity transforms only, such as Transforms.identity("year"). Gravitino does not check them, or the columns, against the Delta transaction log, so keep them in line with the table. Sort orders, distributions, and indexes are rejected.

Table properties

Property	Description	Default	Required
format	Table format. Must be delta.	(none)	Yes
external	Must be true.	(none)	Yes
location	Directory of the Delta table, the one holding _delta_log, on S3, HDFS, Azure storage, GCS, or another Hadoop-compatible file system. The schema and catalog locations are not used.	(none)	Yes

Data type mappings

Delta Lake uses Apache Spark data types. Gravitino types map to them as follows:

Gravitino Type	Delta/Spark Type	Notes
Boolean	BooleanType
Byte	ByteType
Short	ShortType
Integer	IntegerType
Long	LongType
Float	FloatType
Double	DoubleType
Decimal(p, s)	DecimalType(p, s)
String	StringType
Binary	BinaryType
Date	DateType
Timestamp	TimestampNTZType	No timezone, Spark 3.4+
Timestamp_tz	TimestampType	With timezone
List	ArrayType
Map	MapType
Struct	StructType

Troubleshooting

Error	Fix
Gravitino only supports creating external Delta tables	Set "external": "true" in the table properties.
Property 'location' is required for external Delta tables	Set location to the Delta table directory.
ALTER TABLE operations are not supported	Change the table with Delta Lake tools, then drop and register it again.
Purge operation is not supported	Drop the table instead, and delete the data with Delta Lake tools or on storage.

Related

• Generic lakehouse catalog
• Tables and views
• Delta Lake documentation