CLI reference
The table maintenance command-line tool, bin/gravitino-optimizer.sh, evaluates compaction policies,
submits rewrite jobs, and submits statistics jobs. It ships in the server image and runs inside the
server pod. Each command makes one pass and exits; nothing schedules it.
Quick start
The example previews the compaction of {catalog}.{schema}.{table}, assuming the configuration file
from Command-line tool is already written to the pod.
kubectl -n {namespace} exec deploy/{release}-gravitino -- \
env GRAVITINO_MEM=-Xmx512m /opt/gravitino/bin/gravitino-optimizer.sh \
--conf-path /opt/gravitino/logs/table-maintenance/gravitino-optimizer.conf \
--type submit-strategy-jobs \
--identifiers {catalog}.{schema}.{table} \
--strategy-name iceberg_compaction \
--dry-run --limit 10
Every command below takes the same kubectl exec prefix and --conf-path; the examples show only
the options after them. --help lists every command, and --help --type {command} shows one.
Run the tool
The tool runs in the server container, where it reaches the server at http://localhost:8090.
GRAVITINO_MEM caps its heap; without it the tool sizes its heap from the server container's memory
limit, as the server does.
Always pass --conf-path. Without it the tool reads conf/gravitino-optimizer.conf from the image,
whose job submitter is noop-job-submitter: it evaluates and submits nothing, and it carries no
credentials. The configuration file and its keys are described in
Command-line tool.
The tool authenticates to the server with gravitino.optimizer.auth.*:
| Key | Value |
|---|---|
gravitino.optimizer.auth.type | basic or oauth. The default, none, is rejected by the server |
gravitino.optimizer.auth.username | User name, for basic |
gravitino.optimizer.auth.password | Password, for basic |
gravitino.optimizer.auth.oauth.serverUri | OAuth server URI, for oauth |
gravitino.optimizer.auth.oauth.path | Token path on that server, for oauth |
gravitino.optimizer.auth.oauth.credential | Client credential, {client_id}:{client_secret}, for oauth |
gravitino.optimizer.auth.oauth.scope | Scope, for oauth |
Commands
Command (--type) | Required | Optional | Does |
|---|---|---|---|
submit-strategy-jobs | --identifiers, --strategy-name | --dry-run, --limit | Evaluates a policy and submits rewrite jobs |
submit-update-stats-job | --identifiers | --dry-run, --update-mode, --updater-options, --spark-conf | Submits builtin-iceberg-update-stats for each table |
submit-strategy-jobs
Evaluates the named compaction policy against each table in --identifiers, using the statistics
the server already holds, and submits builtin-iceberg-rewrite-data-files for each table the policy
fires on, highest score first.
| Option | Meaning |
|---|---|
--identifiers | Comma-separated tables, catalog.schema.table, or schema.table when gravitinoDefaultCatalog is set |
--strategy-name | The policy name, such as iceberg_compaction; not the policy type or the strategy type |
--dry-run | Prints what would be submitted and submits nothing |
--limit | Most jobs to submit; greater than 0 |
--type submit-strategy-jobs \
--identifiers {catalog}.{schema}.orders,{catalog}.{schema}.customers \
--strategy-name iceberg_compaction \
--limit 10
The rewrite jobs take their parameters from the gravitino.optimizer.jobSubmitterConfig.* entries in
the configuration file, plus the table and the policy's rewriteOptions.
submit-update-stats-job
Submits builtin-iceberg-update-stats for each table, so the statistics a policy evaluates are
current. Run it before submit-strategy-jobs, and again after a compaction, since a rewrite does not
refresh them.
| Option | Meaning |
|---|---|
--identifiers | Comma-separated tables, as for submit-strategy-jobs |
--dry-run | Prints each job's parameters, with credentials masked, and submits nothing |
--update-mode | stats for compaction. Defaults to all, which also writes metrics to a repository the chart does not configure |
--updater-options | Flat JSON map with gravitino_uri, metalake, statistics_updater, and the server credentials |
--spark-conf | Flat JSON map of Spark settings, including the IRC credentials |
Each option falls back to the matching gravitino.optimizer.jobSubmitterConfig.* entry
(update_mode, updater_options, spark_conf), so with the configuration file from
Command-line tool only --identifiers is needed:
--type submit-update-stats-job \
--identifiers {catalog}.{schema}.{table}
When updater_options is set, the tool submits the job with the credentials in it rather than
gravitino.optimizer.auth.*, so it has to carry auth_type and the matching keys
(username and password, or oauth_server_uri, oauth_path, oauth_credential, and
oauth_scope).
Output
| Line | Meaning |
|---|---|
DRY-RUN: strategy=... | A table the policy fired on, with its score; nothing submitted |
SUBMIT: strategy=... jobId=... | A rewrite job submitted |
DRY-RUN: identifier=... jobConfig=... | A statistics job that would be submitted |
SUBMIT: identifier=... jobId=... | A statistics job submitted |
SUMMARY: submit-update-stats-job ... | Totals for a statistics submission |
DRY-RUN: strategy=iceberg-data-compaction identifier=sales.ops.orders score=95 jobTemplate=builtin-iceberg-rewrite-data-files jobOptions={catalog_name=sales, table_identifier=ops.orders}
SUBMIT: strategy=iceberg-data-compaction identifier=sales.ops.orders score=95 jobTemplate=builtin-iceberg-rewrite-data-files jobOptions={catalog_name=sales, table_identifier=ops.orders} jobId=job-1f54c6d3
A job ID from a SUBMIT: line is what Jobs checks status by, and names the job's
staging directory.
Troubleshooting
Command errors come back from the tool at once and name the problem. Evaluation problems print
nothing, which is what makes them confusing. Job failures happen inside Spark, so the reason is in
the job's error.log in its staging directory, /opt/gravitino/logs/jobs/staging/{metalake}/{template}/{job_id}/
(under the logs volume's subPath when one is set), not in the API response or the UI.
Command errors
Invalid --type: command names are kebab-case, such as submit-strategy-jobs.
Specified optimizer config file does not exist: the --conf-path file is not in the pod. Write
it as shown in Command-line tool.
No StrategyHandler class configured for strategy type: iceberg-data-compaction: the
configuration file is missing
gravitino.optimizer.strategyHandler.iceberg-data-compaction.className.
The provided credentials did not support: the tool, or the statistics job calling back, reached
the server without credentials it accepts. Check gravitino.optimizer.auth.*, or the auth keys in
updater_options.
Evaluation prints nothing
No identifiers matched strategy name: --strategy-name takes the policy name. Given the policy
type system_iceberg_compaction or the strategy type iceberg-data-compaction, it matches nothing.
No DRY-RUN or SUBMIT line: the policy did not fire. Read the table's statistics (see
Table maintenance, step 4): either custom-data-file-mse and
custom-delete-file-number are below the policy's thresholds, or they are missing because the
statistics job has not run on the table. Check too that the policy is attached to the table or to its
schema or catalog.
A compacted table still fires: statistics are only as fresh as the last statistics job. Run
submit-update-stats-job again after a compaction.
Job failures
A job's status stays queued: status is polled once a minute, so a job can read as queued until
the next poll. The chart also runs one job at a time, so a job waits behind the one before it.
Every job fails at once and has no staging directory: the server pod has no Spark. The server
log reads gravitino.jobExecutor.local.sparkHome or SPARK_HOME environment variable must be set for Spark jobs. Enable Spark as described in Table maintenance.
Missing Iceberg Spark session extensions: Spark is present but the Iceberg Spark runtime is not.
The chart's bootstrap downloads it; check that the pod could reach Maven Central or the configured
mirror.
The job cannot find the catalog, or looks for default_catalog: warehouse_location must be the
Gravitino catalog name.
Rewrite fails on a multi-level partition: a table combining an identity field with a time, bucket,
or truncate transform, such as PARTITIONED BY (p, days(ts)), fails with:
Cannot translate Spark expression ... day(cast(ts as date)) ... to data source filter
| Partitioning | Rewrite |
|---|---|
p, p, c2 | Works |
p, years(ts), p, months(ts), p, days(ts), p, hours(ts) | Fails |
p, truncate(1, c2), p, bucket(8, id) | Fails |
Compact identity-partitioned tables only.