Automation
A job is a piece of work Gravitino runs on your behalf, and a job template is the reusable definition it runs from. Registering a template once and running it many times is the shape, so the command, its arguments, and its environment live in the catalog rather than in whoever launched it last.
Jobs are how catalog-driven work gets done. Table maintenance uses them, and anything else that should run against metadata Gravitino already knows about can use the same mechanism.
Automate → Overview summarizes the metalake's jobs at a glance: how many are running, how many failed or completed in the last day, and the most recent runs with their templates and status.
Quick start
-
Register a template, or use a built-in one. Automate → Templates lists them. A template names the kind of job, the command or application to run, and the parameters it takes. See Templates.
-
Run it. On Automate → Jobs, click Run job, choose the template, and give values for its parameters. Running a template creates a job. See Jobs.
-
Watch it. Automate → Jobs shows each job's status as it moves through the queue, and finished jobs keep their outcome and output.
Templates and jobs
A template is the definition and a job is one execution of it. Templates are named and reusable; jobs are individual and carry a status. A template with a queued or running job cannot be deleted, and deleting a template removes the records of the jobs already run from it.
The job system runs a job when a caller submits one. It has no schedule of its own.
Permissions
Registering a job template requires REGISTER_JOB_TEMPLATE on the metalake, using one requires
USE_JOB_TEMPLATE on the metalake or on that template, and running a job requires RUN_JOB on the
metalake.
Running a job takes both RUN_JOB and USE_JOB_TEMPLATE for the template it runs from. Altering and
deleting a job template are reserved for the metalake owner and the template owner.
See Privileges for a comprehensive list of privilege grants.
Server configuration
| Setting | Description | Default |
|---|---|---|
gravitino.job.executor | Executor that runs jobs | local |
gravitino.job.stagingDir | Directory holding each job's staging files and logs | /tmp/gravitino/jobs/staging |
gravitino.job.stagingDirKeepTimeInMs | How long a finished job's staging files are kept | 604800000 (7 days) |
gravitino.job.statusPullIntervalInMs | Interval between job status polls | 300000 (5 minutes) |
gravitino.jobExecutor.local.maxRunningJobs | Jobs the local executor runs at once | max(1, min(cores / 2, 10)) |
gravitino.jobExecutor.local.sparkHome | Spark installation the local executor submits Spark jobs with | (none) |
The local executor runs each job as a process inside the Gravitino server's container, sharing its
memory. Running jobs anywhere else means implementing an executor; see
Custom job executor. The rest of the local executor's
properties are in Server configuration.
The local executor submits Spark jobs with bin/spark-submit under sparkHome, and falls back to the
SPARK_HOME environment variable when the property is not set. With neither set, a Spark job fails
to launch. The server image does not include Spark. Setting gravitino.job.spark.bootstrap.enabled
to true in the chart copies Spark 3.5 and the Iceberg Spark runtime into the pod and points
sparkHome at it; see Table maintenance.
The chart sets these from values under gravitino.job: stagingDir, stagingDirKeepTimeInMs,
statusPullIntervalInMs, maxRunningJobs, and sparkHome. By default it puts the staging directory
on the logs volume, sets maxRunningJobs to 1 so concurrent Spark jobs do not exhaust the server
pod's memory, and sets statusPullIntervalInMs to 60000 (1 minute). gravitino.job.executor has
no chart value and defaults to local; set a different executor under
gravitino.additionalConfigItems; see Set other server properties.
Job status is polled rather than pushed, so a job's reported status can lag the real process by a full interval. At the chart's one-minute interval, a job that finished seconds ago can still read as running for a minute, and at the server default of five minutes for longer.
Use the API
Templates and jobs can be registered, listed, run, and cancelled over REST and through the Java and Python clients. The REST reference documents each call for job templates and jobs, with the client code under the Java and Python tabs.