Templates

A job template is the reusable definition a job runs from: the kind of job, what it executes, and the parameters a run can supply. The UI lists the metalake's templates, including the built-in ones the table maintenance service uses, and runs a job from any of them.

Quick start

The example registers a shell template that runs a script against a table, then runs it. Parameters are written as {{name}} in the template and filled in by each run.

UI

1. Start a template. Open Automate → Templates and click + Template.

2. Name it. Enter a Name and choose the Job type, Shell or Spark. A Comment is optional.

3. Define what it runs. Under Execution, enter the Executable, and add Arguments, Environment variables, and the Script. Write each value a run supplies as {{name}}, for example {{table}} as an argument.

4. Add it. Click Add template.

5. Run it. See Jobs for running a template and following the job.

The Templates list shows each template's kind, owner, and comment, including the built-in ones.

REST

1. Register a template. A SHELL template runs a command; a SPARK template submits a Spark application.

   Register a shell template

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/jobs/templates" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
     "jobTemplate": {
       "name": "{template}",
       "jobType": "shell",
       "executable": "/opt/scripts/export.sh",
       "arguments": ["{{table}}"]
     }
   }'

2. Run it. See Jobs for running a template and following the job.

The executable, a shell template's scripts, and a Spark template's jars, files, and archives must be reachable by the Gravitino server, which fetches them when a job runs. It accepts local paths and HTTP, HTTPS, and FTP URLs, and refuses remote URIs that resolve to unsafe addresses unless gravitino.fetchFile.blockUnsafeRemoteUri is turned off. See Server configuration.

Template kinds

Kind	What It Runs
SHELL	A command, with arguments and environment from the template
SPARK	A Spark application, with its main class, jars, and Spark config

A template declares parameters, and each run supplies values for them. That is what lets one template serve many cases rather than needing a near-copy per variation.

A {{name}} placeholder can appear in the executable, the arguments, the environment variables, and the custom fields of either kind, in a shell template's scripts, and in a Spark template's class name, jars, files, archives, and Spark config.

Endpoints

Paths are relative to https://{gravitino_host}/api/metalakes/{metalake}.

Operation	Method	Path
Register a template	POST	/jobs/templates
List templates	GET	/jobs/templates
Get a template	GET	/jobs/templates/{template}
Delete a template	DELETE	/jobs/templates/{template}

Add ?details=true to the list path to get full templates instead of names.

A template cannot be deleted while a job run from it is queued, started, or being cancelled. Deleting a template also removes the records and staging files of the jobs already run from it.

Built-in templates

Three job templates ship with the service, and they are complementary rather than alternatives. A full maintenance pass collects statistics, compacts data files, and then expires the snapshot history that compaction just created.

Job template	What it does
builtin-iceberg-update-stats	Collects file statistics and metrics
builtin-iceberg-rewrite-data-files	Compacts small data files
builtin-iceberg-expire-snapshots	Removes old snapshot metadata

Each can be submitted directly over REST, and the first two are also what the policy-driven workflow submits on your behalf. See Table Maintenance for the policy-driven path.

Update statistics

builtin-iceberg-update-stats reads a table and writes back the statistics and metrics that policies evaluate. Compaction policies read custom-data-file-mse and custom-delete-file-number, so nothing else will fire until this job has run at least once.

Its jobConf is documented in Table Maintenance.

Rewrite data files

builtin-iceberg-rewrite-data-files performs the compaction itself, merging small data files into larger ones. It is what a compaction policy submits when its thresholds are crossed.

In alpha this works only on Iceberg tables where every partition uses an identity transform. Tables combining identity with a time or bucket transform fail during the rewrite, which is covered in CLI Reference.

For the policy that drives it, including threshold tuning, see Iceberg Compaction Policy.

Expire snapshots

builtin-iceberg-expire-snapshots removes old Iceberg snapshots and the metadata files behind them. Without periodic expiration, snapshot JSON files and manifest lists accumulate indefinitely, which slows table operations and wastes storage. Compaction makes this worse, since every rewrite creates a snapshot.

The job calls Iceberg's expire_snapshots stored procedure through Spark SQL.

Property	Value
Name	builtin-iceberg-expire-snapshots
Type	Spark
Version	v1
Main class	org.apache.gravitino.maintenance.jobs.iceberg.IcebergExpireSnapshotsJob

Parameters

catalog_name and table_identifier are required. The rest are optional.

Key	Description	Default
catalog_name	Iceberg catalog name as registered in Spark	Required
table_identifier	Fully qualified table name, such as db.sample	Required
older_than	Expire snapshots older than this yyyy-MM-dd HH:mm:ss timestamp	Five days ago
retain_last	Minimum number of recent snapshots to keep regardless of age	1
stream_results	Streams intermediate delete results when present	Disabled
spark_conf	JSON map of Spark configuration	None

older_than and retain_last work together, and retain_last wins. Setting older_than to yesterday with retain_last at 5 keeps five snapshots even if all five are older than yesterday.

Submitting the job

Submit an expire snapshots job

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -d '{
    "jobTemplateName": "builtin-iceberg-expire-snapshots",
    "jobConf": {
      "catalog_name": "rest_catalog",
      "table_identifier": "db.t1",
      "older_than": "2024-01-01 00:00:00",
      "retain_last": "3",
      "spark_master": "local[2]",
      "spark_executor_instances": "1",
      "spark_executor_cores": "1",
      "spark_executor_memory": "1g",
      "spark_driver_memory": "1g",
      "catalog_type": "rest",
      "catalog_uri": "http://localhost:9001/iceberg",
      "warehouse_location": ""
    }
  }' \
  http://localhost:8090/api/metalakes/test/jobs/runs

Omitting older_than and passing only retain_last is the safer default for a first run, since it bounds the result by count rather than by a date you have to reason about.

The job builds this statement, including only the optional parameters you supplied:

Generated expire snapshots statement

CALL `rest_catalog`.system.expire_snapshots(
  table => 'db.t1',
  older_than => TIMESTAMP '2024-01-01 00:00:00',
  retain_last => 3,
  stream_results => true
)

Verifying the result

Get the job status

curl -sS "http://localhost:8090/api/metalakes/test/jobs/runs/{job_id}" | jq '.job.status'

A successful run reports its status as succeeded. Its output, in output.log in the job's staging directory (see Jobs), includes the counts it removed:

Example job output

Expire Snapshots Results:
  Deleted data files: 12
  Deleted manifest files: 8
  Deleted manifest lists: 3