Templates

A job template is the reusable definition a job runs from: the kind of job, what it executes, and the parameters a run can supply. The UI lists the metalake's templates, including the built-in ones the table maintenance service uses, and runs a job from any of them.

Quick start

The example registers a shell template that runs a script against a table, then runs it. Parameters are written as {{name}} in the template and filled in by each run.

UI

1. Start a template. Open Automate → Templates and click + Template.

2. Name it. Enter a Name and choose the Job type, Shell or Spark. A Comment is optional.

3. Define what it runs. Under Execution, enter the Executable, and add Arguments, Environment variables, and the Script. Write each value a run supplies as {{name}}, for example {{table}} as an argument.

4. Add it. Click Add template.

5. Run it. See Jobs for running a template and following the job.

The Templates list shows each template's kind, owner, and comment, including the built-in ones.

REST

1. Register a template. A shell template runs a command; a spark template submits a Spark application.

   Register a shell template

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/jobs/templates" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
     "jobTemplate": {
       "name": "{template}",
       "jobType": "shell",
       "executable": "/opt/scripts/export.sh",
       "arguments": ["{{table}}"]
     }
   }'

2. Run it. See Jobs for running a template and following the job.

The executable, a shell template's scripts, and a Spark template's jars, files, and archives must be reachable by the Gravitino server, which fetches them when a job runs. It accepts local paths and HTTP, HTTPS, and FTP URLs, and refuses remote URIs that resolve to unsafe addresses unless gravitino.fetchFile.blockUnsafeRemoteUri is turned off. See Server configuration.

Template kinds

Kind	What it runs
shell	A command, with arguments and environment from the template
spark	A Spark application, with its main class, jars, and Spark config

A template declares parameters, and each run supplies values for them. That is what lets one template serve many cases rather than needing a near-copy per variation.

A {{name}} placeholder can appear in the executable, the arguments, the environment variables, and the custom fields of either kind, in a shell template's scripts, and in a Spark template's class name, jars, files, archives, and Spark config.

Built-in templates

Three job templates ship with the server. All three are Spark jobs, so they need Spark in the server pod; see Table maintenance.

Job template	What it does
builtin-sparkpi	Computes pi; a check that Spark jobs run at all
builtin-iceberg-update-stats	Collects the file statistics that compaction policies read
builtin-iceberg-rewrite-data-files	Compacts small data files

Names starting with builtin- are reserved: a template cannot be registered or renamed to such a name, and the built-in templates cannot be altered or deleted.

The two Iceberg templates can be run directly, and are also what a compaction policy submits through the command-line tool. Their parameters are in Table maintenance.

Update statistics

builtin-iceberg-update-stats reads a table's file metadata and writes nine file statistics, including the two compaction policies evaluate, custom-data-file-mse and custom-delete-file-number. No policy fires on a table until this job has run on it, and a compaction does not refresh them.

Rewrite data files

builtin-iceberg-rewrite-data-files performs the compaction itself, merging small data files into larger ones with Iceberg's rewrite_data_files procedure. It works on unpartitioned tables and tables whose partition fields all use identity transforms; a table combining an identity field with a time, bucket, or truncate transform fails during the rewrite. For the policy that drives it, including threshold tuning, see Compaction policy.

Endpoints

Paths are relative to https://{gravitino_host}/api/metalakes/{metalake}.

Operation	Method	Path
Register a template	POST	/jobs/templates
List templates	GET	/jobs/templates
Get a template	GET	/jobs/templates/{template}
Alter a template	PUT	/jobs/templates/{template}
Delete a template	DELETE	/jobs/templates/{template}

Add ?details=true to the list path to get full templates instead of names.

A template cannot be deleted while a job run from it is queued, started, or being cancelled. Deleting a template also removes the records and staging files of the jobs already run from it.