Python client

The Python client calls the REST API from Python code, for notebooks, data science and AI work, and applications such as Spark, PyTorch, TensorFlow and Ray. Each page of the REST reference shows the client code for its call under the Python tab, so this page covers only what every program needs first: installing the package, connecting and signing in. The Python client covers metalakes, catalogs, schemas, tables, views, partitions, table statistics, functions, filesets, models, tags and jobs. Topics, policies and partition statistics are available through the REST API and the Java client only.

Quick start

The example connects to one metalake as a local user and lists its catalogs. It needs Python 3.10 or later.

1. Install the client into a virtual environment.

   Install the Python client

   python3 -m venv ~/gravitino-venv
   ~/gravitino-venv/bin/pip install "apache-gravitino>=1.3.1,<1.4"

2. Connect and list the catalogs.

   list_catalogs.py

   from gravitino import GravitinoClient
   from gravitino.auth.basic_auth_provider import BasicAuthProvider
   
   client = GravitinoClient(
       uri="https://{gravitino_host}",
       metalake_name="{metalake}",
       auth_data_provider=BasicAuthProvider("{user}", "{password}"),
   )
   
   for catalog in client.list_catalogs_info():
       print(catalog.name(), catalog.type().type_name, catalog.provider())

3. Run it with the virtual environment's Python.

   Run the script

   ~/gravitino-venv/bin/python list_catalogs.py

   Each line names a catalog, its type, and its provider. To sign in through your identity provider instead, pass a different auth_data_provider, as described in Sign in.

Install

The package needs Python 3.10 or later. Install 1.3.1 or a later 1.3 release; the 1.3.0 package has no view support. The client works with a server of the same or a later major and minor version, which Get server version reports, so a 1.3 client works with any 1.3 server. Install it into a virtual environment, since Debian and Ubuntu refuse a system-wide pip install.

Install the Python client

python3 -m venv ~/gravitino-venv
~/gravitino-venv/bin/pip install "apache-gravitino>=1.3.1,<1.4"

The package brings in the S3, Azure, Google Cloud and Alibaba Cloud storage libraries that GVFS uses, so the first install takes a few minutes.

Connect

There are two clients. GravitinoAdminClient creates, lists and drops metalakes, and only service admins can use it. GravitinoClient works inside one metalake. The REST reference samples call them admin_client and client.

Create the clients

from gravitino import GravitinoAdminClient, GravitinoClient
from gravitino.auth.basic_auth_provider import BasicAuthProvider

auth = BasicAuthProvider("{user}", "{password}")

admin_client = GravitinoAdminClient(uri="https://{gravitino_host}", auth_data_provider=auth)

client = GravitinoClient(
    uri="https://{gravitino_host}",
    metalake_name="{metalake}",
    auth_data_provider=auth,
)

The URI is the server's address without /api. On connecting, the client checks that its version is compatible with the server's; check_version=False turns the check off.

Imports

The REST reference samples leave out imports. The gravitino package exports the clients, NameIdentifier, Catalog, Schema, Fileset, MetalakeChange, SchemaChange and FilesetChange. Import everything else from its module:

Class	Module
Namespace	gravitino.namespace
CatalogChange	gravitino.api.catalog_change
Column	gravitino.api.rel.column
Types	gravitino.api.rel.types.types
Transforms	gravitino.api.rel.expressions.transforms.transforms
Literals	gravitino.api.rel.expressions.literals.literals
Partitions	gravitino.api.rel.partitions.partitions
TableChange	gravitino.api.rel.table_change
ViewChange	gravitino.api.rel.view_change
SQLRepresentation	gravitino.api.rel.sql_representation
StatisticValues	gravitino.api.stats.statistic_values
FunctionDefinitions	gravitino.api.function.function_definition
FunctionParams	gravitino.api.function.function_param
FunctionImpl	gravitino.api.function.function_impl
FunctionType	gravitino.api.function.function_type
SQLImpl	gravitino.api.function.sql_impl
ModelChange	gravitino.api.model_change
ModelVersionChange	gravitino.api.model_version_change
TagChange	gravitino.api.tag.tag_change
ShellJobTemplate	gravitino.api.job.shell_job_template
JobTemplateChange, ShellTemplateUpdate	gravitino.api.job.job_template_change

Imports for a table sample

from gravitino import NameIdentifier
from gravitino.namespace import Namespace
from gravitino.api.rel.column import Column
from gravitino.api.rel.types.types import Types
from gravitino.api.rel.expressions.transforms.transforms import Transforms

Sign in

Pass the auth_data_provider that matches how the server authenticates. See Authentication for the server side of each.

Provider	Use it for
BasicAuthProvider(user, password)	The admin user and local users
DefaultOAuth2TokenProvider(uri, credential, scope, path)	Users who sign in through your identity provider

Sign in with an OAuth 2 token

from gravitino.auth.default_oauth2_token_provider import DefaultOAuth2TokenProvider

token_provider = DefaultOAuth2TokenProvider(
    uri="https://{oauth_host}",
    credential="{client_id}:{client_secret}",
    scope="{scope}",
    path="{token_path}",
)

client = GravitinoClient(
    uri="https://{gravitino_host}",
    metalake_name="{metalake}",
    auth_data_provider=token_provider,
)

Configure

client_config sets the request timeout, and request_headers adds HTTP headers to every request. An unknown configuration property raises an exception.

Property	Description	Default
gravitino_client_request_timeout	Request timeout, in seconds	10

Set the request timeout

client = GravitinoClient(
    uri="https://{gravitino_host}",
    metalake_name="{metalake}",
    auth_data_provider=auth,
    client_config={"gravitino_client_request_timeout": 60},
)

Errors

A failed call raises an exception named for the problem, such as NoSuchCatalogException, SchemaAlreadyExistsException or ForbiddenException, from gravitino.exceptions.base. The name matches the type field of the REST error response, which each reference page lists under its error status codes.

Filesets from Python

For reading and writing the files behind a fileset with Python file APIs, see GVFS.

Related

• Authentication
• Java client
• GVFS