Skip to main content

Table maintenance

Table maintenance compacts Iceberg tables whose data files have drifted below their target size. You attach a compaction policy to a catalog, schema, or table; a statistics job measures each table; an evaluation compares those measurements with the policy; and a rewrite job merges the small files.

Each of those steps runs when you start it. Nothing schedules them, and a compaction does not refresh the statistics it was decided on, so a compacted table still reads as over threshold until its statistics are collected again.

1. Attach a compaction policy to a catalog, schema, or table. 2. Collect statistics with the builtin-iceberg-update-stats Spark job, which stores them on the table. 3. Evaluate with gravitino-optimizer.sh, which compares the statistics with the policy. 4. Compact with the rewrite job, which merges small data files in object storage. Statistics are stale after a rewrite until step 2 runs again

Table maintenance compacts Iceberg tables only, and only tables that are unpartitioned or whose every partition field uses an identity transform. It does not expire snapshots, remove orphan files, or rewrite manifests; run those from your own engine through the IRC.

The command-line tool, its configuration file, and its properties carry the older name optimizer: bin/gravitino-optimizer.sh, gravitino-optimizer.conf, and gravitino.optimizer.*.

Before you start​

The statistics and rewrite jobs are Spark jobs, and the server runs them with spark-submit inside its own container. The server image does not include Spark, so the chart has to provide it, along with the Iceberg Spark runtime that the jobs load. Add the following to your values file and re-run your helm upgrade --install command.

my-values.yaml
gravitino:
  job:
    spark:
      bootstrap:
        enabled: true
  resources:
    requests:
      memory: 8Gi
    limits:
      memory: 8Gi

With bootstrap enabled, an init container copies Spark 3.5 from the apache/spark image into the pod and downloads iceberg-spark-runtime-3.5_2.12 and iceberg-aws-bundle from Maven Central. The pod therefore needs to reach Docker Hub and Maven Central. For a private registry, set gravitino.job.spark.bootstrap.image.registry; for a Maven mirror, set gravitino.job.spark.bootstrap.icebergJars.download.mavenMirror. A Spark job shares the server container's memory limit, which is why the example raises it from the 4Gi default; a job that exceeds the limit restarts the server. The chart runs one job at a time.

Confirm the two built-in templates are registered before continuing:

List job templates
curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/jobs/templates" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Authorization: Bearer $TOKEN"

The list includes builtin-iceberg-update-stats and builtin-iceberg-rewrite-data-files.

Quick start​

The example compacts table {table} in schema {schema} of Iceberg catalog {catalog}, a table written through the IRC like any other. It needs CREATE_POLICY and APPLY_POLICY for the policy, and RUN_JOB and USE_JOB_TEMPLATE for the jobs; see Privileges.

  1. Create a compaction policy. The UI creates custom policies only, so create this one over REST. An empty content takes the defaults described in Compaction policy; rewriteOptions here lowers the number of files a rewrite needs before it acts.

    Create the policy
    curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/policies" \
      -H "Accept: application/vnd.gravitino.v1+json" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $TOKEN" \
      -d '{
        "name": "iceberg_compaction",
        "comment": "Compact Iceberg tables",
        "policyType": "system_iceberg_compaction",
        "enabled": true,
        "content": {"rewriteOptions": {"min-input-files": "2"}}
      }'

  2. Attach it. A policy on a catalog or schema covers every table beneath it.

    On the table's page in Explore, click Policies in the header, choose iceberg_compaction under Policies, and click Update policies. Catalog and schema pages have the same control.

  3. Collect statistics. Run builtin-iceberg-update-stats for the table. Its parameters, and the credentials it needs to read the table and write the statistics back, are under Job parameters.

    On Automate → Jobs, click Run job, choose builtin-iceberg-update-stats as the Template name, fill in the values from Job parameters, and click Run job.

    Wait for the job to succeed; see Jobs for checking its status.

  4. Read the statistics.

    Get the table's statistics
    curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/objects/table/{catalog}.{schema}.{table}/statistics" \
      -H "Accept: application/vnd.gravitino.v1+json" \
      -H "Authorization: Bearer $TOKEN"

    The response includes custom-data-file-mse and custom-delete-file-number, the two values the policy evaluates. A table of many files a few kilobytes each reads around 1.8e16, far above the default threshold of 405323966463344.

  5. Preview the compaction. Evaluation runs only in the command-line tool. Write its configuration (see Command-line tool), then run a dry run, which evaluates the policy and submits nothing:

    Dry run
    kubectl -n {namespace} exec deploy/{release}-gravitino -- \
      env GRAVITINO_MEM=-Xmx512m /opt/gravitino/bin/gravitino-optimizer.sh \
      --conf-path /opt/gravitino/logs/table-maintenance/gravitino-optimizer.conf \
      --type submit-strategy-jobs \
      --identifiers {catalog}.{schema}.{table} \
      --strategy-name iceberg_compaction \
      --dry-run --limit 10

    A DRY-RUN: line naming builtin-iceberg-rewrite-data-files and a score means the policy fired. No line means the statistics are below the policy's thresholds.

  6. Compact. Submitting from the policy runs the same command without --dry-run, which prints a SUBMIT: line ending in the rewrite job's jobId. Running the rewrite template yourself compacts the table whether or not the policy fired.

    On Automate → Jobs, click Run job, choose builtin-iceberg-rewrite-data-files as the Template name, fill in the values from Job parameters, and click Run job.

  7. Check the result. The Jobs list shows the rewrite's status. When it succeeds, its output.log reports Rewritten data files and Added data files; Jobs gives its location, since the UI does not show a job's output. Then run step 3 again, so the statistics reflect the compacted table, and step 5 again, which now prints no DRY-RUN: line.

Policies​

A compaction policy fires when the table's small-file or delete-file statistic reaches its threshold; Compaction policy explains both values and how to tune them.

Three names in this workflow look interchangeable and are not.

NameExampleWhere it is used
Policy nameiceberg_compactionThe policy's own name, and the value of --strategy-name
Policy typesystem_iceberg_compactionpolicyType when creating the policy
Strategy typeiceberg-data-compactionstrategy.type in the policy, and the strategy handler configuration

--strategy-name takes the policy name. Given either of the other two, the tool reports that no identifiers matched rather than naming the mistake.

Job parameters​

Both templates take the parameters below. Every parameter a template declares has to be supplied, even as an empty string: an omitted one reaches the job as its literal placeholder, such as {{strategy}}, and the job fails. Run job in the UI fills in the defaults for both templates.

The jobs run inside the server's container, so they reach the server at http://localhost:8090 and the IRC at http://localhost:9001/iceberg.

ParameterTemplateValue
catalog_nameBothThe Gravitino catalog name, used as the Spark catalog name
table_identifierBoth{schema}.{table}, without the catalog
catalog_typeBothrest
catalog_uriBothhttp://localhost:9001/iceberg
warehouse_locationBothThe Gravitino catalog name. The IRC selects the catalog by it; left empty, it looks for default_catalog
spark_masterBothlocal[1]
spark_executor_instancesBoth1
spark_executor_coresBoth1
spark_executor_memoryBoth1g
spark_driver_memoryBoth1g, sized within the server container's memory limit
spark_confBothFlat JSON map of Spark settings, including the IRC credentials below
update_modeStatisticsstats for compaction. metrics and all also write to a metrics repository that needs its own JDBC setup
updater_optionsStatisticsFlat JSON map with gravitino_uri, metalake, statistics_updater, and the server credentials below
strategyRewritebinpack, or sort with sort_order
sort_orderRewriteEmpty, or a sort such as id DESC NULLS LAST or zorder(c1,c2)
where_clauseRewriteEmpty, or a predicate limiting the rewrite to matching partitions
optionsRewriteJSON map of Iceberg rewrite options, such as {"min-input-files":"2"}; {} for none

spark_conf and updater_options are JSON strings inside the request's JSON, so their quotes are escaped, which is the most common source of a malformed run.

Credentials​

A job authenticates twice: Spark to the IRC to read and commit the table, and the statistics job to the server to write its results. The job runs as the user those credentials name, so that user needs the privileges to read and modify the table.

ConnectionWhereKeys
Spark to the IRCspark_confBasic: spark.sql.catalog.{catalog}.rest.auth.type = basic, plus .rest.auth.basic.username and .rest.auth.basic.password. OAuth: .rest.auth.type = oauth2, plus .credential, .oauth2-server-uri, and .scope
Statistics to serverupdater_optionsauth_type (basic or oauth); username and password for basic; oauth_server_uri, oauth_path, oauth_credential, and oauth_scope for OAuth

If the catalog vends storage credentials, also set spark.sql.catalog.{catalog}.header.X-Iceberg-Access-Delegation to vended-credentials in spark_conf; otherwise give Spark its own storage credentials there. Job APIs and the UI return both maps with secret values masked.

Command-line tool​

The command-line tool evaluates policies and submits rewrite jobs. It ships in the server image, so it runs inside the server pod through kubectl exec. Each command makes one pass and exits.

Its configuration file has to sit somewhere the pod can write; the logs volume works. Write it from your workstation:

Write the configuration into the pod
kubectl -n {namespace} exec -i deploy/{release}-gravitino -- \
  sh -c 'mkdir -p /opt/gravitino/logs/table-maintenance && cat > /opt/gravitino/logs/table-maintenance/gravitino-optimizer.conf' \
  < gravitino-optimizer.conf
gravitino-optimizer.conf
gravitino.optimizer.gravitinoUri = http://localhost:8090
gravitino.optimizer.gravitinoMetalake = {metalake}
gravitino.optimizer.gravitinoDefaultCatalog = {catalog}
gravitino.optimizer.auth.type = basic
gravitino.optimizer.auth.username = {user}
gravitino.optimizer.auth.password = {password}
gravitino.optimizer.recommender.statisticsProvider = gravitino-statistics-provider
gravitino.optimizer.recommender.strategyProvider = gravitino-strategy-provider
gravitino.optimizer.recommender.tableMetaProvider = gravitino-table-metadata-provider
gravitino.optimizer.recommender.jobSubmitter = gravitino-job-submitter
gravitino.optimizer.strategyHandler.iceberg-data-compaction.className = org.apache.gravitino.maintenance.optimizer.recommender.handler.compaction.CompactionStrategyHandler
gravitino.optimizer.jobSubmitterConfig.catalog_name = {catalog}
gravitino.optimizer.jobSubmitterConfig.catalog_type = rest
gravitino.optimizer.jobSubmitterConfig.catalog_uri = http://localhost:9001/iceberg
gravitino.optimizer.jobSubmitterConfig.warehouse_location = {catalog}
gravitino.optimizer.jobSubmitterConfig.spark_master = local[1]
gravitino.optimizer.jobSubmitterConfig.spark_executor_instances = 1
gravitino.optimizer.jobSubmitterConfig.spark_executor_cores = 1
gravitino.optimizer.jobSubmitterConfig.spark_executor_memory = 1g
gravitino.optimizer.jobSubmitterConfig.spark_driver_memory = 1g
gravitino.optimizer.jobSubmitterConfig.spark_conf = {"spark.sql.catalog.{catalog}.rest.auth.type":"basic","spark.sql.catalog.{catalog}.rest.auth.basic.username":"{user}","spark.sql.catalog.{catalog}.rest.auth.basic.password":"{password}"}
gravitino.optimizer.jobSubmitterConfig.update_mode = stats
gravitino.optimizer.jobSubmitterConfig.updater_options = {"gravitino_uri":"http://localhost:8090","metalake":"{metalake}","statistics_updater":"gravitino-statistics-updater","auth_type":"basic","username":"{user}","password":"{password}"}

Every gravitino.optimizer.jobSubmitterConfig.* entry becomes a parameter of the jobs the tool submits, with the same meaning as in Job parameters. The last two entries let the same file run the statistics job with --type submit-update-stats-job. The tool's own connection takes gravitino.optimizer.auth.type of basic or oauth, since the server accepts no other; OAuth uses gravitino.optimizer.auth.oauth.serverUri, .path, .credential, and .scope. The file holds credentials, so delete it when you are done.

The image's default conf/gravitino-optimizer.conf submits nothing (its job submitter is noop-job-submitter) and names no credentials, which is why every command passes --conf-path. GRAVITINO_MEM caps the tool's heap; without it the tool takes its share of the server container's memory. Every command and option is in the CLI reference.

Limits​

  • The rewrite fails on a table that combines an identity field with a time, bucket, or truncate transform, such as PARTITIONED BY (p, days(ts)). Compact unpartitioned and identity-partitioned tables only.
  • Jobs run in the server container, one at a time, so table maintenance suits tables whose compaction fits in a single Spark driver within that container's memory. Compact larger tables from your own engine through the IRC.
  • Job status is polled every minute, so a finished job can read as running until the next poll. The UI does not show a job's output.
Last updated on