Custom job executor

Introduction

The local job executor that ships with Gravitino runs jobs as a process on the Gravitino server and is intended for testing. Running jobs anywhere else, such as against a distributed scheduler, means implementing your own job executor.

Implement a custom job executor

Gravitino's job system is extensible: you can implement your own job executor to run jobs in a distributed environment. Refer to the interface JobExecutor in the code here.

After you implement your own job executor, register it in the Gravitino server under gravitino.additionalConfigItems in my-values.yaml, with each value as a quoted string; see Set other server properties. For a job executor named airflow:

my-values.yaml

gravitino:
  additionalConfigItems:
    gravitino.job.executor: "airflow"
    gravitino.jobExecutor.airflow.class: "com.example.MyAirflowJobExecutor"

Configure the job executor with additional properties in the same block.

my-values.yaml

gravitino:
  additionalConfigItems:
    gravitino.jobExecutor.airflow.host: "http://localhost:8080"
    gravitino.jobExecutor.airflow.username: "myuser"
    gravitino.jobExecutor.airflow.password: "mypassword"

These properties will be passed to the airflow job executor when it is instantiated.