Skip to main content

Templates

A job template is the reusable definition a job runs from: the kind of job, what it executes, and the parameters a run can supply. The UI lists the metalake's templates, including the built-in ones the table maintenance service uses, and runs a job from any of them.

Quick Start​

1. Register a template. A SHELL template runs a command; a SPARK template submits a Spark application. Parameters are written as {{name}} in the template and filled in by each run.

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/jobs/templates" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "jobTemplate": {
    "name": "{template}",
    "jobType": "shell",
    "executable": "/opt/scripts/export.sh",
    "arguments": ["{{table}}"]
  }
}'

executable must be reachable by the Gravitino server, which accepts local paths and HTTP, HTTPS, FTP, and FTPS URLs.

2. Run it. See Jobs for running a template and following the job.

Template Kinds​

KindWhat It Runs
SHELLA command, with arguments and environment from the template
SPARKA Spark application, with its main class, jars, and Spark config

A template declares parameters, and each run supplies values for them. That is what lets one template serve many cases rather than needing a near-copy per variation.

Endpoints​

Paths are relative to https://{gravitino_host}/api/metalakes/{metalake}.

OperationMethodPath
Register a templatePOST/jobs/templates
List templatesGET/jobs/templates
Get a templateGET/jobs/templates/{template}
Delete a templateDELETE/jobs/templates/{template}

Add ?details=true to the list path to get full templates instead of names.

Built-in Templates​

Three job templates ship with the service, and they are complementary rather than alternatives. A full maintenance pass collects statistics, compacts data files, and then expires the snapshot history that compaction just created.

Job templateWhat it does
builtin-iceberg-update-statsCollects file statistics and metrics
builtin-iceberg-rewrite-data-filesCompacts small data files
builtin-iceberg-expire-snapshotsRemoves old snapshot metadata

Each can be submitted directly over REST, and the first two are also what the policy-driven workflow submits on your behalf. See Table Maintenance for the policy-driven path.

Update Statistics​

builtin-iceberg-update-stats reads a table and writes back the statistics and metrics that policies evaluate. Compaction policies read custom-data-file-mse and custom-delete-file-number, so nothing else will fire until this job has run at least once.

Its jobConf is documented in Table Maintenance.

Rewrite Data Files​

builtin-iceberg-rewrite-data-files performs the compaction itself, merging small data files into larger ones. It is what a compaction policy submits when its thresholds are crossed.

In alpha this works only on Iceberg tables where every partition uses an identity transform. Tables combining identity with a time or bucket transform fail during the rewrite, which is covered in CLI Reference.

For the policy that drives it, including threshold tuning, see Iceberg Compaction Policy.

Expire Snapshots​

builtin-iceberg-expire-snapshots removes old Iceberg snapshots and the metadata files behind them. Without periodic expiration, snapshot JSON files and manifest lists accumulate indefinitely, which slows table operations and wastes storage. Compaction makes this worse, since every rewrite creates a snapshot.

The job calls Iceberg's expire_snapshots stored procedure through Spark SQL.

PropertyValue
Namebuiltin-iceberg-expire-snapshots
TypeSpark
Versionv1
Main classorg.apache.gravitino.maintenance.jobs.iceberg.IcebergExpireSnapshotsJob

Parameters​

catalog_name and table_identifier are required. The rest are optional.

KeyDescriptionDefault
catalog_nameIceberg catalog name as registered in SparkRequired
table_identifierFully qualified table name, such as db.sampleRequired
older_thanExpire snapshots older than this yyyy-MM-dd HH:mm:ss timestampFive days ago
retain_lastMinimum number of recent snapshots to keep regardless of age1
stream_resultsStreams intermediate delete results when presentDisabled
spark_confJSON map of Spark configurationNone

older_than and retain_last work together, and retain_last wins. Setting older_than to yesterday with retain_last at 5 keeps five snapshots even if all five are older than yesterday.

Submitting the Job​

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -d '{
    "jobTemplateName": "builtin-iceberg-expire-snapshots",
    "jobConf": {
      "catalog_name": "rest_catalog",
      "table_identifier": "db.t1",
      "older_than": "2024-01-01 00:00:00",
      "retain_last": "3",
      "spark_master": "local[2]",
      "spark_executor_instances": "1",
      "spark_executor_cores": "1",
      "spark_executor_memory": "1g",
      "spark_driver_memory": "1g",
      "catalog_type": "rest",
      "catalog_uri": "http://localhost:9001/iceberg",
      "warehouse_location": ""
    }
  }' \
  http://localhost:8090/api/metalakes/test/jobs/runs

Omitting older_than and passing only retain_last is the safer default for a first run, since it bounds the result by count rather than by a date you have to reason about.

The job builds this statement, including only the optional parameters you supplied:

CALL `rest_catalog`.system.expire_snapshots(
  table => 'db.t1',
  older_than => TIMESTAMP '2024-01-01 00:00:00',
  retain_last => 3,
  stream_results => true
)

Verifying the Result​

curl -sS "http://localhost:8090/api/metalakes/test/jobs/runs/{job_id}?includeOutput=true" | jq '.job'

A successful run reports its status as SUCCEEDED, and its output includes the counts it removed:

Expire Snapshots Results:
  Deleted data files: 12
  Deleted manifest files: 8
  Deleted manifest lists: 3
Last updated on