Python client development

This page covers building, testing, and releasing the Gravitino Python client. For using the client in your own code, see Python client.

The client lives in clients/client-python and publishes to PyPI as apache-gravitino. Gradle drives every step, including the Python toolchain itself, so the commands below are Gradle tasks rather than direct pip or python invocations.

Quick Start

1. Install the client and its development dependencies. Gradle provisions a Miniforge environment at the Python version the root project declares, so no local virtual environment is needed.

./gradlew :clients:client-python:pipInstall

2. Run the unit tests.

./gradlew :clients:client-python:unitTests

3. Run the full check. The test task formats, lints, and runs both unit and integration tests.

./gradlew :clients:client-python:test

Integration tests start a real Gravitino server, so the first full run takes noticeably longer than the unit tests alone. Skip them with -PskipITs, or skip unit tests with -PskipTests.

Requirements

Python 3.10 is the minimum, and the package is classified for 3.10, 3.11, and 3.12. The interpreter Gradle uses comes from the root project's pythonVersion, not from whatever python3 resolves to on your machine.

Dependencies are split across three files. Runtime dependencies live in requirements.txt and are read directly by setup.py. Development dependencies live in requirements-dev.txt and are exposed as the dev extra. Lance integration dependencies live in requirements-lance.txt and are exposed as the lance extra, though the same pins also appear in the development file so that a default integration run works without installing the extra separately.

Gradle Tasks

Task	What it does
pipInstall	Installs the client in editable mode with the dev extra
black	Formats gravitino, tests, and scripts
pylint	Lints the same three directories
build	Runs scripts/generate_version.py after linting
unitTests	Runs the unit suite under coverage
integrationTest	Starts a Gravitino server, runs the integration suite, stops it
test	Formats, lints, and runs both suites
unitCoverageReport	Writes an HTML coverage report for the unit suite
integrationCoverageReport	Writes an HTML coverage report for the integration suite
lanceRayMatrixTest	Runs the Lance integration test against several lance-ray versions
doc	Builds the Sphinx API documentation
distribution	Builds the source distribution for PyPI
deploy	Uploads the distribution to PyPI
clean	Removes build output, coverage data, and generated files

Tests

Both suites use unittest rather than pytest, run through coverage. Unit tests are in tests/unittests and integration tests in tests/integration. Gradle sets PYTHONPATH to the client directory for both, which is why running the suites outside Gradle needs that variable set by hand.

Integration Tests

The integration suite needs a built Gravitino distribution, because Gradle starts the server by running distribution/package/bin/gravitino.sh. Build the distribution before running them.

Startup is confirmed by polling the server's metrics endpoint for a version field, with a thirty second timeout, so a server that starts slowly on a cold machine can fail the run without the tests themselves being at fault.

The suite runs against an already-running server by default, since Gradle sets START_EXTERNAL_GRAVITINO and manages the lifecycle itself rather than letting each test class start one.

Coverage output lands in tests/integration/htmlcov and tests/unittests/htmlcov.

Lance Matrix Tests

lanceRayMatrixTest exercises tests/integration/test_lance_ray.py against several lance-ray versions, each in its own cached virtual environment under build/lance-ray-matrix. The default version set tracks the compatibility matrix on the Lance integration page.

Three properties control it. Use -PlanceRayVersions with a comma-separated list to override the version set, -PlanceRayKeepGoing to continue past a failing version, and -PlanceRayPython to choose the bootstrap interpreter. The bootstrap interpreter only provisions the per-version environments and does not affect what lance-ray sees at runtime.

Formatting and Linting

Two toolchains are configured and they do not overlap cleanly. Gradle runs black for formatting and pylint for linting. The pre-commit configuration runs ruff for both linting and formatting, with a line length of 88 and double quotes.

Running one after the other can produce churn, so decide which path you are on before committing. Gradle is the path CI exercises, since the test task depends on black and pylint.

API Documentation

The published Python API documentation is built with Sphinx using autodoc and autosummary, themed with furo.

./gradlew :clients:client-python:doc

Output lands in docs/build inside the client directory. The Sphinx configuration reads the project version and author out of setup.py by regular expression, so a change to how those fields are written in setup.py breaks the documentation build even when the package still installs.

Releasing

distribution builds a source distribution and deploy uploads it with twine, reading the password from the TWINE_PASSWORD environment variable.

The client's README.md is generated, not maintained. The distribution task deletes any existing README.md, regenerates it from docs/how-to-use-python-client.md by stripping the frontmatter and rewriting relative documentation and image links into absolute ones, uses it as the PyPI long description, and deletes it again afterward. Edits to README.md are lost on the next build, so changes to the PyPI landing page belong in the source page instead.

The package version is declared in setup.py, and scripts/generate_version.py writes gravitino/version.ini during the build task.