Skip to main content

Functions

A function is a piece of logic registered in the catalog so engines can find it and people can govern it. Registering one puts it in the same hierarchy as tables, with a name, an owner, tags, and permissions.

Gravitino stores the definition rather than executing it. Whether a function runs, and how, is a question for the engine that reads it.

Quick start​

The example registers the scalar function add_one in the schema public of the catalog sales, with a SQL implementation for Trino, then looks at it and tags it certified. The tag must already exist; see Tags. Registering, altering, and dropping a function go through the API; the UI shows functions and tags them.

  1. Register the function. The UI does not register functions. Use step 1 of the REST tab, or the Java or Python client.

  2. Find it. In Explore, open sales and then public. The schema's functions are listed with its tables and views, with their tags and policies.

  3. Look at it. Click add_one. The Overview tab shows its properties, owner, and when and by whom it was registered. The Definitions tab shows each definition's signature, here add_one(x: integer) => integer, and each implementation's language, runtime, body or class, and resources.

  4. Tag it. Back in public, select the add_one row, click + Tags, choose certified under Additional tags, and click Update tags.

The function's own page does not show its tags, function type, determinism flag, or comment. Read those through the API.

An engine reading through Gravitino resolves the function by name and runs the implementation for its own runtime.

The function model​

Function types​

TypeReturns
SCALAROne value per row
AGGREGATEOne value per group
TABLEA set of rows, for table-valued operations

The type is declared when the function is registered and tells an engine how to call it.

Determinism​

A function is marked deterministic or not. A deterministic function returns the same result for the same arguments every time, which lets an engine cache, reorder, or eliminate calls to it. Marking a function deterministic when it is not, such as one that reads the clock or a random source, invites an engine to optimize in ways that change the answer.

Gravitino records the flag and does not verify it.

Definitions and implementations​

A function has two levels beneath it.

A definition is an overload: one parameter list and one return type. A function with several definitions is the same name accepting different arguments, the way round(x) and round(x, digits) are one function in most engines. A TABLE function's definition declares its output columns, in returnColumns, instead of a return type.

An implementation is how a definition is expressed for a given engine. Each declares a language and a runtime, so one definition can carry a SQL version for Trino and a Java version for Spark, and each engine uses the one it can execute.

LanguageCarries
SQLThe SQL text of the function body
PYTHONA handler entrypoint, and optionally inline code
JAVAA class name

Runtimes are SPARK and TRINO. Java and Python implementations can also carry resources, such as a jar or a package the runtime needs to load.

Examples by function type​

Which registered functions an engine can call depends on its connector. The Trino connector runs scalar functions that have a SQL implementation for the TRINO runtime. The Spark connector runs scalar and aggregate functions that have a Java implementation for the SPARK runtime.

Function typeTrino connectorSpark connector
SCALARSQL implementation, TRINOJAVA implementation, SPARK
AGGREGATENot availableJAVA implementation, SPARK
TABLENot availableNot available

Gravitino registers, lists, tags, and grants on every type and language. Table functions and Python implementations are not yet run by either connector in 1.3; that is a limit of the connectors, not of the function model. The connector pages cover the details: Trino and Spark.

The examples register into the schema public of the catalog sales, as in the quick start.

Scalar function​

mask_email hides the local part of an email address. One definition carries two implementations: SQL for Trino, and a Java class for Spark. Each engine picks the implementation for its runtime.

Register a scalar function for Trino and Spark
curl -sS -X POST "https://{gravitino_host}/api/metalakes/example/catalogs/sales/schemas/public/functions" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "name": "mask_email",
  "functionType": "scalar",
  "deterministic": true,
  "comment": "Replaces the part of an email address before the @ with ***",
  "definitions": [{
    "parameters": [{"name": "email", "dataType": "string"}],
    "returnType": "string",
    "impls": [
      {
        "language": "SQL",
        "runtime": "TRINO",
        "sql": "concat('"'"'***'"'"', substr(email, strpos(email, '"'"'@'"'"')))"
      },
      {
        "language": "JAVA",
        "runtime": "SPARK",
        "className": "com.example.udf.MaskEmail",
        "resources": {"jars": ["s3://{bucket}/udfs/example-udfs.jar"]}
      }
    ]
  }]
}'
Call it from Trino
SELECT sales.public.mask_email('jane.doe@example.com');
-- ***@example.com
Call it from Spark
SELECT customer_id, sales.public.mask_email(email) AS email
FROM sales.public.customers;

The Spark class implements Spark's UnboundFunction and binds to a ScalarFunction. The connector does not fetch the jar named in resources; put it on the Spark classpath with --jars or spark.jars.

Aggregate function​

geo_mean returns the geometric mean of a group of values. An aggregate function has a return type like a scalar one, and the engine calls it once per group.

Register an aggregate function for Spark
curl -sS -X POST "https://{gravitino_host}/api/metalakes/example/catalogs/sales/schemas/public/functions" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "name": "geo_mean",
  "functionType": "aggregate",
  "deterministic": true,
  "comment": "Geometric mean of positive values",
  "definitions": [{
    "parameters": [{"name": "x", "dataType": "double"}],
    "returnType": "double",
    "impls": [{
      "language": "JAVA",
      "runtime": "SPARK",
      "className": "com.example.udf.GeoMean",
      "resources": {"jars": ["s3://{bucket}/udfs/example-udfs.jar"]}
    }]
  }]
}'
Call it from Spark
SELECT region, sales.public.geo_mean(growth_rate) AS avg_growth
FROM sales.public.quarterly_results
GROUP BY region;

Here the class's bind returns a Spark AggregateFunction rather than a ScalarFunction.

Table function​

split_tags turns a comma-separated string into rows. A table function declares its output columns in returnColumns and has no returnType. The implementation here is Python, with the handler and its code stored inline.

Register a table function
curl -sS -X POST "https://{gravitino_host}/api/metalakes/example/catalogs/sales/schemas/public/functions" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
  "name": "split_tags",
  "functionType": "table",
  "deterministic": true,
  "comment": "One row per tag in a comma-separated list",
  "definitions": [{
    "parameters": [{"name": "tags", "dataType": "string"}],
    "returnColumns": [
      {"name": "tag", "dataType": "string"},
      {"name": "position", "dataType": "integer"}
    ],
    "impls": [{
      "language": "PYTHON",
      "runtime": "SPARK",
      "handler": "split_tags",
      "codeBlock": "def split_tags(tags):\n    for i, t in enumerate(tags.split(\",\")):\n        yield (t.strip(), i)"
    }]
  }]
}'

The function is listed, tagged, and governed like any other, and its definition is readable through the API for tools that run Python table functions themselves. Neither connector exposes it to queries in 1.3.

Permissions​

Registering a function requires REGISTER_FUNCTION on its schema or above. Calling a function from an engine requires EXECUTE_FUNCTION, and altering a registered function requires MODIFY_FUNCTION.

Granting at a wider scope covers everything beneath it. Dropping a function is reserved for the metalake owner and the object owner.

See Privileges for a comprehensive list of privilege grants.

Use the API​

Functions can be registered, listed, altered, and dropped over REST and through the Java and Python clients. The REST reference documents each call, with the client code under the Java and Python tabs.

Last updated on