Skip to main content

Lance REST API

The Lance REST API is the Lance REST catalog protocol, served by the Lance REST service. Lance clients such as lance-spark, lance-ray, and pylance use it to create, register, describe, alter, and drop Lance tables whose metadata Gravitino holds.

Base URL and authentication​

Every path in this reference is relative to https://{lance_host}/lance, where {lance_host} is the host the service is published on; see HTTPS. Each call needs the same credentials the Gravitino server accepts: a bearer token from the identity provider, or a local user's name and password.

Example​

The script walks the Lance REST API from end to end: it gets an access token, creates a namespace hierarchy, creates a table, inspects it, changes its columns, and drops it. It needs curl, jq, and Python with pyarrow, and assumes the Lance REST service is published on the Ingress as described in Client access.

The service accepts the same credentials as the Gravitino server. The script requests a token from the identity provider with the client credentials grant; any client the Gravitino server trusts works. The catalog namespace carries the S3 credentials that the service and Lance clients use to reach the table data.

lance-rest-walkthrough.sh
#!/usr/bin/env bash
set -euo pipefail

# Set inputs
LANCE_URI="https://{lance_host}/lance"
TOKEN_ENDPOINT="{token_endpoint}"
CLIENT_ID="{client_id}"
CLIENT_SECRET="{client_secret}"
CATALOG="lance_catalog"
SCHEMA="demo"
TABLE="articles"
TABLE_LOCATION="s3://{bucket}/lance/${SCHEMA}/${TABLE}"
NS="${CATALOG}%24${SCHEMA}"        # catalog$schema, with $ URL-encoded
TBL="${NS}%24${TABLE}"             # catalog$schema$table

# Get an access token from the identity provider
TOKEN=$(curl -sSf "$TOKEN_ENDPOINT" \
  -d grant_type=client_credentials \
  -d client_id="$CLIENT_ID" \
  -d client_secret="$CLIENT_SECRET" | jq -r '.access_token')
AUTH="Authorization: Bearer ${TOKEN}"

# Create the catalog namespace, which is a lakehouse generic catalog
curl -sSf -X POST "${LANCE_URI}/v1/namespace/${CATALOG}/create" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{
    "mode": "exist_ok",
    "properties": {
      "lance.storage.aws_region": "{aws_region}",
      "lance.storage.aws_access_key_id": "{aws_access_key_id}",
      "lance.storage.aws_secret_access_key": "{aws_secret_access_key}"
    }
  }' | jq .

# Create the schema namespace
curl -sSf -X POST "${LANCE_URI}/v1/namespace/${NS}/create" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"mode": "exist_ok"}' | jq .

# List the namespaces in the catalog
curl -sSf "${LANCE_URI}/v1/namespace/${CATALOG}/list" -H "$AUTH" | jq .

# Write the table schema as an Arrow IPC stream
python3 -c '
import sys, pyarrow as pa
schema = pa.schema([("id", pa.int64()), ("title", pa.string()), ("body", pa.string())])
pa.ipc.new_stream(sys.stdout.buffer, schema).close()
' > schema.arrow

# Create the table from the schema
curl -sSf -X POST "${LANCE_URI}/v1/table/${TBL}/create" \
  -H "$AUTH" \
  -H 'Content-Type: application/vnd.apache.arrow.stream' \
  -H "x-lance-table-location: ${TABLE_LOCATION}" \
  --data-binary @schema.arrow | jq .

# List the tables in the schema
curl -sSf "${LANCE_URI}/v1/namespace/${NS}/table/list" -H "$AUTH" | jq .

# Describe the table: its location, schema, version, and the storage options a client uses
curl -sSf -X POST "${LANCE_URI}/v1/table/${TBL}/describe" \
  -H "$AUTH" -H 'Content-Type: application/json' -d '{}' | jq .

# Rename a column
curl -sSf -X POST "${LANCE_URI}/v1/table/${TBL}/alter_columns" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"alterations": [{"path": "body", "rename": "content"}]}' | jq .

# Drop a column
curl -sSf -X POST "${LANCE_URI}/v1/table/${TBL}/drop_columns" \
  -H "$AUTH" -H 'Content-Type: application/json' \
  -d '{"columns": ["title"]}' | jq .

# Drop the table, deleting its metadata and data
curl -sSf -X POST "${LANCE_URI}/v1/table/${TBL}/drop" \
  -H "$AUTH" -H 'Content-Type: application/json' -d '{}' | jq .

Supported operations​

The service supports the namespace and table operations below. Index operations are not supported.

OperationDescriptionHTTP MethodEndpoint Pattern
CreateNamespaceCreate a new Lance namespacePOST/lance/v1/namespace/{id}/create
ListNamespacesList all namespaces under a parent namespaceGET/lance/v1/namespace/{parent}/list
DescribeNamespaceRetrieve detailed information about a specific namespacePOST/lance/v1/namespace/{id}/describe
DropNamespaceDelete a namespacePOST/lance/v1/namespace/{id}/drop
NamespaceExistsCheck whether a namespace existsPOST/lance/v1/namespace/{id}/exists
ListTablesList all tables in a namespaceGET/lance/v1/namespace/{id}/table/list
CreateTableCreate a new table in a namespacePOST/lance/v1/table/{id}/create
DescribeTableDescribe an existing Lance tablePOST/lance/v1/table/{id}/describe
DropTableDelete a table including both metadata and dataPOST/lance/v1/table/{id}/drop
TableExistsCheck whether a table existsPOST/lance/v1/table/{id}/exists
RegisterTableRegister an existing Lance table to a namespacePOST/lance/v1/table/{id}/register
DeregisterTableUnregister a table from a namespace (metadata only, data remains)POST/lance/v1/table/{id}/deregister
DeclareTableDeclare a table and store the metadata without touching lance table data.POST/lance/v1/table/{id}/declare
AlterColumnsRename columns of a tablePOST/lance/v1/table/{id}/alter_columns
DropColumnsDrop columns from a tablePOST/lance/v1/table/{id}/drop_columns

For more details, refer to the Lance REST API specification.

Operation details​

Mode values are case-insensitive; the examples use lowercase.

Namespace operations​

CreateNamespace takes a mode: create fails if the namespace exists, exist_ok succeeds either way, and overwrite replaces an existing namespace.

DropNamespace deletes everything under the namespace, child namespaces and tables, metadata and Lance data files alike. It cannot be undone.

Table operations​

CreateTable writes a new Lance dataset from the schema in the request. RegisterTable records a dataset that already exists, without moving it. DeclareTable records the table's metadata without writing any data. The version in a CreateTable response is always null, meaning the latest version.

DropTable deletes the metadata and the data files. DeregisterTable deletes only the metadata and leaves the data files in place.

Column operations​

AlterColumns renames columns only. Each entry in alterations names a column in path and its new name in rename; an entry that sets data_type, nullable, or virtual_column is rejected. DropColumns takes the column names to drop in columns. Both return the table version after the change.

Namespaces and identifiers​

A Lance namespace has exactly two levels: a catalog, then a schema. Tables live only in a schema.

Namespace hierarchy
metalake
└── lance_catalog      catalog namespace, a generic lakehouse catalog
    └── sales          schema namespace
        └── orders     table
  • A catalog namespace is a generic lakehouse catalog. Create it through the service, as the example does, or through the Gravitino API.
  • A schema namespace can be created only once its catalog exists.
  • A namespace below the schema level, such as lance_catalog/sales/archive, is rejected.

Identifiers join the levels with $, which must be URL-encoded as %24 in a request path:

Identifier forms
Levels:         lance_catalog, sales, orders
Identifier:     lance_catalog$sales$orders
In a URL path:  lance_catalog%24sales%24orders

Client examples​

The examples create namespaces and tables from the shell, Java, and Python. They expect an access token in the TOKEN environment variable, obtained as in the example.

Create namespaces and tables
# Create a catalog-level namespace
# mode: "create" | "exist_ok" | "overwrite" for create namespace/table; mode: "create" | "overwrite" for register table
curl -X POST https://{lance_host}/lance/v1/namespace/lance_catalog/create \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "id": ["lance_catalog"],
    "mode": "create"
  }'

# Create a schema namespace
# Note: %24 is URL-encoded '$' character used as delimiter
curl -X POST https://{lance_host}/lance/v1/namespace/lance_catalog%24schema/create \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "id": ["lance_catalog", "schema"],
    "mode": "create"
  }'

# Register an existing table
curl -X POST https://{lance_host}/lance/v1/table/lance_catalog%24schema%24table01/register \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "id": ["lance_catalog", "schema", "table01"],
    "location": "s3://{bucket}/lance_catalog/schema/table01",
    "mode": "create"
  }'

# Declare a table
curl -X POST https://{lance_host}/lance/v1/table/lance_catalog%24schema%24table04/declare \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "id": ["lance_catalog", "schema", "table04"],
    "location": "s3://{bucket}/lance_catalog/schema/table04"
  }'
  
# Create a table with schema, the schema is inferred from the Arrow IPC file
curl -X POST \
     "https://{lance_host}/lance/v1/table/lance_catalog%24schema%24table03/create" \
     -H 'Content-Type: application/vnd.apache.arrow.stream' \
     -H "Authorization: Bearer $TOKEN" \
     -H "x-lance-table-location: s3://{bucket}/lance_catalog/schema/table03" \
     -H "x-lance-table-properties: {}" \
     --data-binary "@${ARROW_FILE}"
Last updated on