Skip to main content

Spark connector: user-defined functions

The Spark connector loads user-defined functions registered in Gravitino. Once a function is registered, Spark SQL can call it by name, with no CREATE FUNCTION statement in Spark.

The connector runs only Java implementations registered with the SPARK runtime. SQL and Python implementations registered in Gravitino are not visible in Spark.

Prerequisites​

  1. The Spark connector is configured and the catalog is reachable.
  2. The function is registered in Gravitino with at least one definition that has a Java implementation for the SPARK runtime; see Register a function.
  3. The jar holding the function class is on the Spark classpath, for example through --jars or spark.jars.

Java UDF requirements​

The class named in the implementation's className must implement Spark's org.apache.spark.sql.connector.catalog.functions.UnboundFunction interface, described in the Spark API documentation. It needs a public no-argument constructor and must be on both the driver and executor classpath.

Call functions in Spark SQL​

Call a registered function by its three-part name, {catalog}.{schema}.{function}.

Call a function by its full name
-- Call a scalar function
SELECT {catalog}.{schema}.add_one(42);

-- Use in a query
SELECT id, {catalog}.{schema}.add_one(value) AS incremented
FROM {catalog}.{schema}.{table};

After USE, the short name works.

Set the default catalog and schema
USE {catalog};
USE {schema};
SELECT add_one(42);

Discover functions​

SHOW FUNCTIONS lists every function with at least one implementation for the SPARK runtime, whatever its language. Functions registered only for other runtimes, such as TRINO, are neither listed nor loadable. Calling a function uses its first SPARK implementation, which must be Java; register Spark functions with Java implementations only.

List functions in a schema
SHOW FUNCTIONS IN {catalog}.{schema};
Last updated on