Lance REST service

Overview

The Lance REST service provides a RESTful interface for managing Lance datasets through HTTP endpoints. Introduced in Gravitino version 1.1.0, this service enables seamless interaction with Lance datasets for data operations and metadata management.

The service implements the Lance REST API specification. For detailed specification documentation, see the official Lance REST documentation.

About Lance

Lance is a modern columnar data format designed for AI/ML workloads. It provides:

High-performance vector search: Native support for similarity search on high-dimensional embeddings
Columnar storage: Optimized for analytical queries and machine learning pipelines
Fast random access: Efficient row-level operations unlike traditional columnar formats
Version control: Built-in dataset versioning and time-travel capabilities
Incremental updates: Append and update data without full rewrites

Architecture

The Lance REST service acts as a bridge between Lance datasets and applications:

Lance REST service architecture
┌─────────────────┐
│   Applications   │
│  (Python/Java)   │
└────────┬────────┘
         │ HTTP/REST
         ▼
┌─────────────────┐
│   Lance REST     │
│    Service       │ 
└────────┬────────┘
         │ 
         ▼ Gravitino internal API or HTTP client
┌─────────────────┐
│ Gravitino Server │ 
│(Metadata Backend)│
└────────┬────────┘
         │ File System Operations
         ▼
┌─────────────────┐
│  Lance Datasets  │ 
│ (S3/GCS/Local)   │
└─────────────────┘

Key Features:

Full compliance with Lance REST API specification
Can run standalone or integrated with Gravitino server
Support for namespace and table management
Index creation and management capabilities (Index operations are not supported in version 1.1.0)
Metadata stored in Gravitino for unified governance

Supported operations

The Lance REST service provides comprehensive support for namespace management, table management, and index operations. The table below lists all supported operations:

Operation	Description	HTTP Method	Endpoint Pattern
CreateNamespace	Create a new Lance namespace	POST	`/lance/v1/namespace/{id}/create`
ListNamespaces	List all namespaces under a parent namespace	GET	`/lance/v1/namespace/{parent}/list`
DescribeNamespace	Retrieve detailed information about a specific namespace	POST	`/lance/v1/namespace/{id}/describe`
DropNamespace	Delete a namespace	POST	`/lance/v1/namespace/{id}/drop`
NamespaceExists	Check whether a namespace exists	POST	`/lance/v1/namespace/{id}/exists`
ListTables	List all tables in a namespace	GET	`/lance/v1/namespace/{id}/table/list`
CreateTable	Create a new table in a namespace	POST	`/lance/v1/table/{id}/create`
DescribeTable	Describe an existing Lance table	POST	`/lance/v1/table/{id}/describe`
DropTable	Delete a table including both metadata and data	POST	`/lance/v1/table/{id}/drop`
TableExists	Check whether a table exists	POST	`/lance/v1/table/{id}/exists`
RegisterTable	Register an existing Lance table to a namespace	POST	`/lance/v1/table/{id}/register`
DeregisterTable	Unregister a table from a namespace (metadata only, data remains)	POST	`/lance/v1/table/{id}/deregister`
DeclareTable	Declare a table and store the metadata without touching lance table data.	POST	`/lance/v1/table/{id}/declare`

For more details, refer to the Lance REST API specification.

Operation details

Some operations have specific behaviors and modes. Below are important details to consider:

Mode values are parsed case-insensitively. The examples below use lowercase values as the REST-style canonical form.

Namespace operations

CreateNamespace supports three modes:

create: Fails if namespace already exists
exist_ok: Succeeds even if namespace exists
overwrite: Replaces existing namespace

DropNamespace behavior:

Recursively deletes all child namespaces and tables
Deletes both metadata and Lance data files
Operation is irreversible

Table operations

RegisterTable vs CreateTable:

RegisterTable: Links existing Lance datasets into Gravitino catalog without data movement
CreateTable: Creates a new Lance table with a schema and writes metadata files

The version field of CreateTable response is always null, which stands for the latest version.

DropTable vs DeregisterTable:

DropTable: Permanently deletes metadata and data files from storage
DeregisterTable: Removes metadata from Gravitino but preserves Lance data files

Deployment

Run with Gravitino server

To enable the Lance REST service within Gravitino server, set the following properties through the chart values in my-values.yaml and re-run the helm upgrade --install command:

Configuration Property	Description	Default Value	Required
`gravitino.auxService.names`	Auxiliary services to run. Include `lance-rest` to enable Lance REST service	iceberg-rest,lance-rest	Yes
`gravitino.lance-rest.classpath`	Classpath for Lance REST service, relative to Gravitino home directory	lance-rest-server/libs	Yes
`gravitino.lance-rest.httpPort`	Port number for Lance REST service	9101	No
`gravitino.lance-rest.host`	Hostname for Lance REST service	0.0.0.0	No
`gravitino.lance-rest.namespace-backend`	Namespace metadata backend (currently only `gravitino` is supported)	gravitino	Yes
`gravitino.lance-rest.gravitino-uri`	Gravitino server URI. Not required in auxiliary mode.	http://localhost:8090	No
`gravitino.lance-rest.gravitino-metalake`	Gravitino metalake name (required when namespace-backend is `gravitino`)	(none)	Yes

The chart sets gravitino.auxService.names from the value of the same name, and each gravitino.lance-rest.* property above from the matching value under gravitino.lanceRest: classpath, host, httpPort, namespaceBackend, gravitinoUri and gravitinoMetalake. The chart's defaults match the table and already include lance-rest in gravitino.auxService.names, so the metalake is usually the only value to set:

my-values.yaml
gravitino:
  lanceRest:
    gravitinoMetalake: "{metalake}"

In auxiliary mode, gravitino.lance-rest.gravitino-uri is not required because the Lance REST service uses Gravitino's internal API.

Authentication to the Gravitino Server

The Lance REST service makes its own requests to the Gravitino server. Those requests must carry credentials, otherwise the Gravitino server rejects them and every Lance operation fails. The service reaches Gravitino with oauth2 or simple, of which Datastrato Enterprise accepts only oauth2. Configure it to match the Gravitino server:

Configuration Property	Description	Default Value	Required
`gravitino.lance-rest.gravitino-auth-type`	Auth type used to reach the Gravitino server. Supported values: `simple`, `oauth2`	`simple`	No
`gravitino.lance-rest.gravitino-oauth2.server-uri`	OAuth server URI	(none)	Yes, for `oauth2`
`gravitino.lance-rest.gravitino-oauth2.credential`	Credential used to request the OAuth token	(none)	Yes, for `oauth2`
`gravitino.lance-rest.gravitino-oauth2.token-path`	Path on the OAuth server used to request the token	(none)	Yes, for `oauth2`
`gravitino.lance-rest.gravitino-oauth2.scope`	Scope of the requested OAuth token	(none)	Yes, for `oauth2`

These properties have no dedicated chart value. Set them under gravitino.additionalConfigItems as quoted strings; see Set Other Server Properties.

my-values.yaml
gravitino:
  additionalConfigItems:
    gravitino.lance-rest.gravitino-auth-type: "oauth2"
    gravitino.lance-rest.gravitino-oauth2.server-uri: "{oauth_server_uri}"
    gravitino.lance-rest.gravitino-oauth2.credential: "{client_id}:{client_secret}"
    gravitino.lance-rest.gravitino-oauth2.token-path: "{token_path}"
    gravitino.lance-rest.gravitino-oauth2.scope: "{scope}"

These properties control how the service authenticates to the Gravitino server. They do not change how callers authenticate to the Lance REST service itself.

Run standalone

To run the Lance REST service independently of the Gravitino server (the Gravitino server must already be running):

Start the standalone service

{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start

Configure the service by editing {GRAVITINO_HOME}/conf/gravitino-lance-rest-server.conf or passing command-line arguments:

Configuration Property	Description	Default Value	Required
`gravitino.lance-rest.namespace-backend`	Namespace metadata backend	gravitino	Yes
`gravitino.lance-rest.gravitino-uri`	Gravitino server URI	http://localhost:8090	Yes
`gravitino.lance-rest.gravitino-metalake`	Gravitino metalake name	(none)	Yes
`gravitino.lance-rest.httpPort`	Service port number	9101	No
`gravitino.lance-rest.host`	Service hostname	0.0.0.0	No

In standalone deployments, you only need to configure gravitino.lance-rest.gravitino-metalake, gravitino.lance-rest.gravitino-uri, and other properties can use their default values.

Run with Docker

Launch the Lance REST service using Docker (the Gravitino server must already be running):

Run the service in Docker
docker run -d --name lance-rest-service -p 9101:9101 \
  -e LANCE_REST_GRAVITINO_METALAKE_NAME={metalake_name} \
  -e LANCE_REST_GRAVITINO_URI=http://{gravitino_host}:{gravitino_port} \
  apache/gravitino-lance-rest:latest

Access the service at http://localhost:9101.

Environment Variables:

Environment Variable	Configuration Property	Required	Default Value
`LANCE_REST_NAMESPACE_BACKEND`	`gravitino.lance-rest.namespace-backend`	Yes	`gravitino`
`LANCE_REST_GRAVITINO_METALAKE_NAME`	`gravitino.lance-rest.gravitino-metalake`	Yes	(none)
`LANCE_REST_GRAVITINO_URI`	`gravitino.lance-rest.gravitino-uri`	Yes	`http://localhost:8090`
`LANCE_REST_HOST`	`gravitino.lance-rest.host`	No	`0.0.0.0`
`LANCE_REST_PORT`	`gravitino.lance-rest.httpPort`	No	`9101`

Configuration tips.

Required: Set LANCE_REST_GRAVITINO_METALAKE_NAME to your Gravitino metalake name
Conditional: Update LANCE_REST_GRAVITINO_URI if Gravitino server is not on localhost in the Docker instance.
Optional: Other variables can use default values unless you have specific requirements

Usage guidelines

When using Lance REST service with Gravitino backend, keep the following considerations in mind:

Prerequisites

A running Gravitino server with a metalake

Namespace hierarchy

Gravitino follows a three-level hierarchy: catalog → schema → table. When creating namespaces or tables:

Parent must exist: Before creating lance_catalog/schema, ensure lance_catalog catalog exists in Gravitino metalake.
Two-level limit: Create namespace lance_catalog/schema, but not lance_catalog/schema/sub_schema.
Table placement: Tables can only be created under lance_catalog/schema, not at catalog level.

Example namespace hierarchy
metalake
└── lance_catalog (catalog - create via REST)
    └── schema (namespace - create via REST)
        └── table01 (table - create via REST)

Delimiter convention

The Lance REST API uses $ as the default delimiter to separate namespace levels in URIs. When making HTTP requests:

URL Encoding Required: $ must be URL-encoded as %24
Example: lance_catalog$schema$table01 becomes lance_catalog%24schema%24table01 in URLs

Delimiter representations
Namespace path:     lance_catalog.schema.table01
URI representation: lance_catalog$schema$table01  
URL encoded:        lance_catalog%24schema%24table01

Important limitations.

Supports only two levels of namespaces before tables
Tables cannot be nested deeper than schema level
Parent catalog must be created in Gravitino before using Lance REST API
Namespace deletion is recursive and irreversible

Examples

The following examples demonstrate how to interact with Lance REST service using different programming languages and tools.

Prerequisites:

Gravitino server is running with Lance REST service enabled.
A metalake has been created in Gravitino.

Shell
Java
Python

Create namespaces and tables
# Create a catalog-level namespace
# mode: "create" | "exist_ok" | "overwrite" for create namespace/table; mode: "create" | "overwrite" for register table
curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog/create \
  -H 'Content-Type: application/json' \
  -d '{
    "id": ["lance_catalog"],
    "mode": "create"
  }'

# Create a schema namespace
# Note: %24 is URL-encoded '$' character used as delimiter
curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog%24schema/create \
  -H 'Content-Type: application/json' \
  -d '{
    "id": ["lance_catalog", "schema"],
    "mode": "create"
  }'

# Register an existing table
curl -X POST http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table01/register \
  -H 'Content-Type: application/json' \
  -d '{
    "id": ["lance_catalog", "schema", "table01"],
    "location": "/tmp/lance_catalog/schema/table01",
    "mode": "create"
  }'

# Declare a table
curl -X POST http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table04/declare \
  -H 'Content-Type: application/json' \
  -d '{
    "id": ["lance_catalog", "schema", "table04"],
    "location": "/tmp/lance_catalog/schema/table04"
  }'
  
# Create a table with schema, the schema is inferred from the Arrow IPC file
curl -X POST \
     "http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table03/create" \
     -H 'Content-Type: application/vnd.apache.arrow.stream' \
     -H "x-lance-table-location: /tmp/lance_catalog/schema/table03" \
     -H "x-lance-table-properties: {}" \
     --data-binary "@${ARROW_FILE}"    

Create namespaces and tables
// Add dependency: implementation("org.lance:lance-namespace-core:0.4.5")

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.lance.namespace.client.apache.ApiClient;
import org.lance.namespace.client.apache.api.TableApi;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

// Initialize allocator and namespace connection
private final BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);

Map<String, String> props = new HashMap<>();
props.put("uri", "http://localhost:9101/lance");
props.put("delimiter", "$");

LanceNamespace ns = LanceNamespace.connect("rest", props, allocator);

// Create catalog namespace
CreateNamespaceRequest createCatalogNsRequest = new CreateNamespaceRequest();
createCatalogNsRequest.addIdItem("lance_catalog");
createCatalogNsRequest.setMode("create");
ns.createNamespace(createCatalogNsRequest);

// Create schema namespace
CreateNamespaceRequest createSchemaNsRequest = new CreateNamespaceRequest();
createSchemaNsRequest.addIdItem("lance_catalog");
createSchemaNsRequest.addIdItem("schema");
createSchemaNsRequest.setMode("create");
ns.createNamespace(createSchemaNsRequest);

// Register a table
RegisterTableRequest registerTableRequest = new RegisterTableRequest();
registerTableRequest.setLocation("/tmp/lance_catalog/schema/table01");
registerTableRequest.setId(Arrays.asList("lance_catalog", "schema", "table01"));
registerTableRequest.setMode("create");
ns.registerTable(registerTableRequest);

// Declare a table
DeclareTableRequest declareTableRequest = new DeclareTableRequest();
declareTableRequest.setLocation("/tmp/lance_catalog/schema/table02");
declareTableRequest.setId(Arrays.asList("lance_catalog", "schema", "table02"));
ns.declareTable(declareTableRequest);

// Create a table with schema inferred from Arrow IPC file.
// For REST create API, location/properties are passed via headers.
org.apache.arrow.vector.types.pojo.Schema schema =
        new org.apache.arrow.vector.types.pojo.Schema(
                Arrays.asList(
                        Field.nullable("id", new ArrowType.Int(32, true)),
                        Field.nullable("value", new ArrowType.Utf8())));
byte[] body = ArrowUtils.generateIpcStream(schema);
TableApi tableApi = new TableApi(new ApiClient().setBasePath("http://localhost:9101/lance"));
Map<String, String> createTableHeaders = new HashMap<>();
createTableHeaders.put("x-lance-table-location", "/tmp/lance_catalog/schema/table03");
createTableHeaders.put("x-lance-table-properties", "{}");
tableApi.createTable(
    "lance_catalog$schema$table03", body, "$", "create", createTableHeaders);

Create namespaces and tables
# Install: pip install lance-namespace==0.4.5

import lance_namespace as ln
import requests

# Connect to Lance REST service
ns = ln.connect("rest", {"uri": "http://{lance_rest_host}:9101/lance"})

# Create catalog namespace
create_catalog_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog"])
catalog = ns.create_namespace(create_catalog_ns_request)

# Create schema namespace
create_schema_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog", "schema"])
schema = ns.create_namespace(create_schema_ns_request)

# Register a table
register_table_request = ln.RegisterTableRequest(
    id=['lance_catalog', 'schema', 'table01'],
    location='/tmp/lance_catalog/schema/table01'
)
ns.register_table(register_table_request)

# Declare a table
declare_table_request = ln.DeclareTableRequest(
    id=['lance_catalog', 'schema', 'table02'],
    location='/tmp/lance_catalog/schema/table02'
)
ns.declare_table(declare_table_request)

# Create a table with schema inferred from Arrow IPC file.
# For REST create API, location/properties are passed via headers.
with open('schema.ipc', 'rb') as f:
    body = f.read()

response = requests.post(
    "http://{lance_rest_host}:9101/lance/v1/table/lance_catalog%24schema%24table03/create",
    params={"delimiter": "$", "mode": "create"},
    headers={
        "Content-Type": "application/vnd.apache.arrow.stream",
        "x-lance-table-location": "/tmp/lance_catalog/schema/table03",
        "x-lance-table-properties": "{}",
    },
    data=body,
    timeout=30,
)
response.raise_for_status()

Integration with Lance REST

To use the Lance REST service with Apache Spark, Ray and other engines, refer to lance-rest-integration for more details.

Overview​

About Lance​

Architecture​

Supported operations​

Operation details​

Namespace operations​

Table operations​

Deployment​

Run with Gravitino server​

Run standalone​

Run with Docker​

Usage guidelines​

Prerequisites​

Namespace hierarchy​

Delimiter convention​

Examples​

Integration with Lance REST​