Custom job executor

The local job executor that ships with Gravitino runs each job as a process inside the server's container, sharing its memory. Running jobs anywhere else, such as on a distributed scheduler, means implementing your own job executor.

Implement a custom job executor

Gravitino's job system is extensible: you can implement your own job executor to run jobs in a distributed environment. An executor implements the JobExecutor interface.

After you implement your own job executor, register it in the Gravitino server under gravitino.additionalConfigItems in my-values.yaml, with each value as a quoted string; see Set other server properties. For a job executor named airflow:

my-values.yaml

gravitino:
  additionalConfigItems:
    gravitino.job.executor: "airflow"
    gravitino.jobExecutor.airflow.class: "com.example.MyAirflowJobExecutor"

Configure the job executor with additional properties in the same block.

my-values.yaml

gravitino:
  additionalConfigItems:
    gravitino.jobExecutor.airflow.host: "https://{airflow_host}"
    gravitino.jobExecutor.airflow.username: "{airflow_user}"
    gravitino.jobExecutor.airflow.password: "{airflow_password}"

Gravitino passes these properties to the Airflow job executor when it creates it.