Jobs

A job is one run of a job template. The UI lists the metalake's jobs with the template each ran from and its status, and a job can be started from a template or cancelled there.

Quick start

The example runs a job from a template and checks how it went. Each run supplies values for the template's parameters.

UI

1. Start a run. Open Automate → Jobs and click Run job.

2. Choose the template. Choose the Template name. The template's executable, arguments, and Spark configuration appear under Execution, with a placeholder for each value the run must supply.

3. Supply the values. Fill in the required values, marked with an asterisk, and any optional ones.

4. Run it. Click Run job.

5. Check its status. The Jobs list shows each run with its job ID, template, status, who started it, when, and how long it ran. The UI does not show a job's output; step 3 of the REST tab says where the server writes it.

REST

1. Run a template. jobConf supplies values for the template's parameters.

   Run a job template

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/jobs/runs" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{"jobTemplateName": "{template}", "jobConf": {"{parameter}": "{value}"}}'

   The response carries the job's id.

2. Check its status.

   Get the job status

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/jobs/runs/{job_id}" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

3. Read its output. The API does not return a job's output. The local executor writes it to output.log and error.log in the job's staging directory on the server, {staging_dir}/{metalake}/{template}/{job_id}/. The chart puts the staging directory at /opt/gravitino/logs/jobs/staging.

   Read the job output

   kubectl exec -n {namespace} deploy/{release}-gravitino -c gravitino -- \
     cat /opt/gravitino/logs/jobs/staging/{metalake}/{template}/{job_id}/output.log

Job status

Status	Meaning
queued	Waiting to be executed
started	Currently executing
succeeded	Finished successfully
failed	Finished with an error
cancelling	Cancellation requested, not yet complete
cancelled	Cancellation complete

Status is polled from the executor rather than pushed, so it can lag the real process by up to gravitino.job.statusPullIntervalInMs. A job that reads as running for a few minutes after it finished is lagging, not stuck. See Automation.

Cancel a job

Cancellation is a request rather than an instant, which is why cancelling and cancelled are separate. A job that finishes before the cancellation lands stays in its finished state. In the UI, Cancel job on the job's row in Automate → Jobs sends the same request.

Cancel a job

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/jobs/runs/{job_id}" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Authorization: Bearer $TOKEN"

Endpoints

Paths are relative to https://{gravitino_host}/api/metalakes/{metalake}.

Operation	Method	Path
Run a template	POST	/jobs/runs
List jobs	GET	/jobs/runs
List jobs for one template	GET	/jobs/runs?jobTemplateName={template}
Get a job	GET	/jobs/runs/{job_id}
Cancel a job	POST	/jobs/runs/{job_id}

Running a job takes RUN_JOB and USE_JOB_TEMPLATE for its template. See Automation.