Skip to main content

Compaction policy

A compaction policy, policy type system_iceberg_compaction, sets the thresholds at which table maintenance compacts an Iceberg table and the options the rewrite runs with. Attach it to a catalog, schema, or table; a policy on a catalog or schema covers every table beneath it.

Quick start​

  1. Create the policy. The UI creates custom policies only, so create a compaction policy over REST. An empty content takes every default in Policy content.

    Create a compaction policy
    curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/policies" \
      -H "Accept: application/vnd.gravitino.v1+json" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $TOKEN" \
      -d '{
        "name": "iceberg_compaction",
        "comment": "Compact Iceberg tables",
        "policyType": "system_iceberg_compaction",
        "enabled": true,
        "content": {"rewriteOptions": {"min-input-files": "5"}}
      }'

  2. Attach it.

    On the catalog, schema, or table page in Explore, click Policies in the header, choose iceberg_compaction under Policies, and click Update policies.

Creating a policy takes CREATE_POLICY on the metalake, and attaching it takes APPLY_POLICY; see Privileges.

Policy content​

Every field is optional.

FieldDefaultMeaning
minDataFileMse405323966463344Fires when custom-data-file-mse reaches this value, in bytes². Must be >= 0
minDeleteFileNumber1Fires when custom-delete-file-number reaches this value. Must be >= 0
dataFileMseWeight1Weight of custom-data-file-mse in the score. Must be >= 0
deleteFileNumberWeight100Weight of custom-delete-file-number in the score. Must be >= 0
maxPartitionNum50Most partitions one evaluation selects for rewrite. Must be > 0
rewriteOptions{}Iceberg rewrite_data_files options passed to the rewrite job, such as min-input-files

Every compaction policy also carries two fixed properties that cannot be set: strategy.type is iceberg-data-compaction and job.template-name is builtin-iceberg-rewrite-data-files.

How the policy decides​

The two values the policy reads are table statistics written by the builtin-iceberg-update-stats job:

  • custom-data-file-mse is the mean squared shortfall of each data file below 128 MiB. A file at or above 128 MiB contributes nothing, so many small files push it up quickly.
  • custom-delete-file-number counts position and equality delete files.

The policy fires when either reaches its threshold:

Trigger
custom-data-file-mse >= minDataFileMse || custom-delete-file-number >= minDeleteFileNumber

When several tables or partitions qualify, the score orders them:

Score
custom-data-file-mse × dataFileMseWeight + custom-delete-file-number × deleteFileNumberWeight

Tune the policy​

Set minDataFileMse from a target file size and the shortfall you tolerate:

Threshold
minDataFileMse = (target file size in bytes × ratio)²

A ratio between 0.1 and 0.2 suits most tables. The default uses 128 MiB (134217728 bytes) and 0.15, giving 405323966463344.

minDeleteFileNumber of 1 compacts a table as soon as it has a delete file, which keeps merge-on-read tables fast to read at the cost of frequent rewrites. Raise it to compact less often. Raising deleteFileNumberWeight puts tables with delete files ahead of tables that only have small files.

A starting point for production is the defaults plus these rewriteOptions:

OptionValueEffect
target-file-size-bytes134217728Writes 128 MiB files, matching the size the statistics measure
min-input-files5Skips a file group with fewer than five files
delete-file-threshold1Rewrites any data file that has a delete file against it
Last updated on