Iceberg tables

An Iceberg table in Datastrato Enterprise belongs to an Iceberg catalog and is created either by an engine through the Iceberg REST service or through the Gravitino API. The page is the reference for defining one: format versions, column types, partitioning and sort orders, table properties, the changes a table accepts, and dropping it. How the format itself works, from snapshots to maintenance, is covered in How Iceberg works.

Quick start

The example runs in Spark SQL connected to the Iceberg REST service as the catalog gravitino_irc, set up as described in Spark. It creates a format version 3 table partitioned by day, sets its sort order and its delete mode, and reads its properties back.

1. Create the table. format-version is fixed when the table is created.

   Create a partitioned version 3 table

   CREATE NAMESPACE IF NOT EXISTS gravitino_irc.sales;
   
   CREATE TABLE gravitino_irc.sales.orders (
     id BIGINT,
     customer STRING,
     amount DECIMAL(10,2),
     order_ts TIMESTAMP
   ) USING iceberg
   PARTITIONED BY (days(order_ts))
   TBLPROPERTIES (
     'format-version' = '3',
     'write.delete.mode' = 'merge-on-read'
   );

2. Set the sort order. New data files are written sorted by customer, which narrows the files a filter on customer has to read.

   Sort new data by customer

   ALTER TABLE gravitino_irc.sales.orders WRITE ORDERED BY customer;

3. Read the definition back.

   Show the table's properties and layout

   SHOW TBLPROPERTIES gravitino_irc.sales.orders;
   DESCRIBE TABLE EXTENDED gravitino_irc.sales.orders;

Format versions

The format version decides which features a table can use. It is set with the format-version property when the table is created, and an engine can read a table only if its own Iceberg library supports that version. The server is built with Apache Iceberg 1.11.0.

Version	Adds	Status in this release
1	Analytic tables: appends, and overwrites of whole files	Supported
2	Row-level deletes through position and equality delete files, which merge-on-read needs	Supported. The default for new tables
3	Deletion vectors, row lineage, column default values, and the types variant, geometry, geography, timestamp_ns, timestamptz_ns and unknown	Supported
4	Changes still being defined	Can be created, but the spec is not final

Use version 2 for tables that every engine in use must read. Use version 3 for the newer types, or for deletion vectors, which make merge-on-read deletes cheaper to read; check first that each engine's Iceberg library reads version 3. Row-level deletes explains delete files and deletion vectors.

Column types

Engines define columns in Iceberg types directly. Through the Gravitino API, columns use Gravitino types, which map to Iceberg types as follows:

Gravitino type	Apache Iceberg type
Struct	Struct
Map	Map
List	Array
Boolean	Boolean
Integer	Integer
Long	Long
Float	Float
Double	Double
String	String
Date	Date
Time(6)	Time
Timestamp(6)	TimestampType withoutZone
Timestamp_tz(6)	TimestampType withZone
Timestamp(9)	TimestampNanoType withoutZone
Timestamp_tz(9)	TimestampNanoType withZone
Decimal	Decimal
Fixed	Fixed
Binary	Binary
UUID	UUID
Variant	Variant
Null	Unknown
Geometry	Geometry
Geography	Geography

Variant, Geometry, Geography, the nanosecond timestamps and Null need format version 3. Gravitino Null maps to the Iceberg unknown type, a null-only placeholder for a column whose type is not yet known. It must be nullable, and schema evolution can later change it to a concrete type.

Apache Iceberg has no equivalent of the Gravitino Varchar, Fixedchar, Byte, Short and Union types. Any Iceberg type without a Gravitino equivalent is shown as the Gravitino external type. The Gravitino API does not set column default values, although format version 3 supports them; set them from an engine.

Partitioning and sort order

Engines set partitioning and sort order in their own SQL, such as PARTITIONED BY (days(order_ts)) and WRITE ORDERED BY customer in Spark. Through the Gravitino API, a table takes these forms.

Partitions

Supported transforms	Not supported
IdentityTransform, BucketTransform, TruncateTransform, YearTransform, MonthTransform, DayTransform, HourTransform	ApplyTransform, RangeTransform, ListTransform, and BucketTransform over more than one field

Sort orders

A sort order takes a FieldReference, or a FunctionExpression using bucket, truncate, year, month, day or hour. For bucket and truncate, the first argument must be an integer literal and the second a field reference.

Distributions

HashDistribution distributes written data by partition key, and RangeDistribution by partition key or sort key. EvenDistribution is not supported. Without a distribution, a table with a sort order uses RangeDistribution and a partitioned table uses HashDistribution.

Iceberg tables have no indexes.

Table properties

Any Iceberg table property can be set when creating an Iceberg table, except the reserved ones below. Reserved properties cannot be set through Gravitino. Immutable properties cannot be changed once set.

Configuration item	Description	Default value	Required	Reserved	Immutable
location	Iceberg location for table storage.	(none)	No	No	Yes
provider	The storage provider for table storage.	(none)	No	No	Yes
format	The format of table storage.	(none)	No	No	Yes
format-version	The Iceberg table format version, 1 to 4. See Format versions.	2	No	No	Yes
comment	The table comment; use the comment field in table meta instead.	(none)	No	Yes	No
creator	The table creator.	(none)	No	Yes	No
current-snapshot-id	The snapshot represents the current state of the table.	(none)	No	Yes	No
cherry-pick-snapshot-id	Selecting a specific snapshot in a merge operation.	(none)	No	Yes	No
sort-order	Iceberg table sort order; use SortOrder in table meta instead.	(none)	No	Yes	No
identifier-fields	The identifier fields for defining the table.	(none)	No	Yes	No
write.distribution-mode	Defines distribution of write data; use distribution in table meta instead.	(none)	No	Yes	No

These Iceberg properties are the ones most often set, and are described where they matter in How Iceberg works:

Property	Controls
write.delete.mode, write.update.mode, write.merge.mode	Copy-on-write or merge-on-read for row-level changes
write.target-file-size-bytes	The data file size writers aim for
commit.retry.num-retries	How many times a writer retries a commit that lost a race
history.expire.max-snapshot-age-ms	How old a snapshot must be before snapshot expiry removes it
write.metadata.delete-after-commit.enabled	Whether old metadata files are deleted after each commit
write.metadata.previous-versions-max	How many old metadata files are kept when they are deleted

Supported changes

An Iceberg table accepts these changes through the Gravitino API: RenameTable, SetProperty, RemoveProperty, UpdateComment, AddColumn, DeleteColumn, RenameColumn, UpdateColumnType, UpdateColumnPosition, UpdateColumnNullability and UpdateColumnComment. Column changes change only metadata; see Schema and partition evolution.

A new column is added at position LAST unless another position is given. Adding a non-nullable column, or changing a nullable column to non-nullable, can break readers and writers of existing data. See Tables and views for the calls.

Drop and purge

Dropping a table removes it from the catalog and leaves its files in storage. Purging removes the files as well. See Tables and views.

Related

• How Iceberg works
• Iceberg catalog
• Iceberg REST catalog service
• Table maintenance