Python client
The Python client calls the REST API from Python code, for notebooks, data science and AI work, and applications such as Spark, PyTorch, TensorFlow and Ray. Each page of the REST reference shows the client code for its call under the Python tab, so this page covers only what every program needs first: installing the package, connecting and signing in. The Python client covers metalakes, catalogs, schemas, tables, views, partitions, table statistics, functions, filesets, models, tags and jobs. Topics, policies and partition statistics are available through the REST API and the Java client only.
Quick start
The example connects to one metalake as a local user and lists its catalogs. It needs Python 3.10 or later.
-
Install the client into a virtual environment.
Install the Python clientpython3 -m venv ~/gravitino-venv
~/gravitino-venv/bin/pip install "apache-gravitino>=1.3.1,<1.4" -
Connect and list the catalogs.
list_catalogs.pyfrom gravitino import GravitinoClient
from gravitino.auth.basic_auth_provider import BasicAuthProvider
client = GravitinoClient(
uri="https://{gravitino_host}",
metalake_name="{metalake}",
auth_data_provider=BasicAuthProvider("{user}", "{password}"),
)
for catalog in client.list_catalogs_info():
print(catalog.name(), catalog.type().type_name, catalog.provider()) -
Run it with the virtual environment's Python.
Run the script~/gravitino-venv/bin/python list_catalogs.pyEach line names a catalog, its type, and its provider. To sign in through your identity provider instead, pass a different
auth_data_provider, as described in Sign in.
Install
The package needs Python 3.10 or later. Install 1.3.1 or a later 1.3 release; the 1.3.0 package has
no view support. The client works with a server of the same or a later major and minor version,
which Get server version reports, so a 1.3 client works with any 1.3
server. Install it into a virtual environment, since Debian and Ubuntu refuse a system-wide
pip install.
python3 -m venv ~/gravitino-venv
~/gravitino-venv/bin/pip install "apache-gravitino>=1.3.1,<1.4"
The package brings in the S3, Azure, Google Cloud and Alibaba Cloud storage libraries that GVFS uses, so the first install takes a few minutes.
Connect
There are two clients. GravitinoAdminClient creates, lists and drops metalakes, and only service
admins can use it. GravitinoClient works inside one metalake. The REST reference samples call them
admin_client and client.
from gravitino import GravitinoAdminClient, GravitinoClient
from gravitino.auth.basic_auth_provider import BasicAuthProvider
auth = BasicAuthProvider("{user}", "{password}")
admin_client = GravitinoAdminClient(uri="https://{gravitino_host}", auth_data_provider=auth)
client = GravitinoClient(
uri="https://{gravitino_host}",
metalake_name="{metalake}",
auth_data_provider=auth,
)
The URI is the server's address without /api. On connecting, the client checks that its version is
compatible with the server's; check_version=False turns the check off.
Imports
The REST reference samples leave out imports. The gravitino package exports the clients,
NameIdentifier, Catalog, Schema, Fileset, MetalakeChange, SchemaChange and
FilesetChange. Import everything else from its module:
| Class | Module |
|---|---|
Namespace | gravitino.namespace |
CatalogChange | gravitino.api.catalog_change |
Column | gravitino.api.rel.column |
Types | gravitino.api.rel.types.types |
Transforms | gravitino.api.rel.expressions.transforms.transforms |
Literals | gravitino.api.rel.expressions.literals.literals |
Partitions | gravitino.api.rel.partitions.partitions |
TableChange | gravitino.api.rel.table_change |
ViewChange | gravitino.api.rel.view_change |
SQLRepresentation | gravitino.api.rel.sql_representation |
StatisticValues | gravitino.api.stats.statistic_values |
FunctionDefinitions | gravitino.api.function.function_definition |
FunctionParams | gravitino.api.function.function_param |
FunctionImpl | gravitino.api.function.function_impl |
FunctionType | gravitino.api.function.function_type |
SQLImpl | gravitino.api.function.sql_impl |
ModelChange | gravitino.api.model_change |
ModelVersionChange | gravitino.api.model_version_change |
TagChange | gravitino.api.tag.tag_change |
ShellJobTemplate | gravitino.api.job.shell_job_template |
JobTemplateChange, ShellTemplateUpdate | gravitino.api.job.job_template_change |
from gravitino import NameIdentifier
from gravitino.namespace import Namespace
from gravitino.api.rel.column import Column
from gravitino.api.rel.types.types import Types
from gravitino.api.rel.expressions.transforms.transforms import Transforms
Sign in
Pass the auth_data_provider that matches how the server authenticates. See
Authentication for the server side of each.
| Provider | Use it for |
|---|---|
BasicAuthProvider(user, password) | The admin user and local users |
DefaultOAuth2TokenProvider(uri, credential, scope, path) | Users who sign in through your identity provider |
from gravitino.auth.default_oauth2_token_provider import DefaultOAuth2TokenProvider
token_provider = DefaultOAuth2TokenProvider(
uri="https://{oauth_host}",
credential="{client_id}:{client_secret}",
scope="{scope}",
path="{token_path}",
)
client = GravitinoClient(
uri="https://{gravitino_host}",
metalake_name="{metalake}",
auth_data_provider=token_provider,
)
Configure
client_config sets the request timeout, and request_headers adds HTTP headers to every request. An
unknown configuration property raises an exception.
| Property | Description | Default |
|---|---|---|
gravitino_client_request_timeout | Request timeout, in seconds | 10 |
client = GravitinoClient(
uri="https://{gravitino_host}",
metalake_name="{metalake}",
auth_data_provider=auth,
client_config={"gravitino_client_request_timeout": 60},
)
Errors
A failed call raises an exception named for the problem, such as NoSuchCatalogException,
SchemaAlreadyExistsException or ForbiddenException, from gravitino.exceptions.base. The name
matches the type field of the REST error response, which each reference page lists under its error
status codes.
Filesets from Python
For reading and writing the files behind a fileset with Python file APIs, see GVFS.