Lance tables

A generic lakehouse catalog manages Lance tables: tables whose format property is lance. Gravitino records each table's columns and location, and creates, alters, and deletes the Lance dataset at that location. Lance clients such as lance-spark and lance-ray reach the same tables through the Lance REST service. How the format stores a table, and how to fill one with embeddings, is covered in How Lance works.

Quick start

The example creates the Lance table orders in the schema sales of the generic lakehouse catalog lakehouse_catalog, set up as in the generic lakehouse catalog Quick start. Gravitino writes an empty Lance dataset under the schema's location.

1. Create the table. format is required. Without a location, the table lands at {catalog_location}/sales/orders.

   Create a Lance table

   curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $TOKEN" \
     -d '{
       "name": "orders",
       "comment": "Orders",
       "columns": [
         {"name": "id", "type": "long", "nullable": false},
         {"name": "amount", "type": "double", "nullable": true}
       ],
       "properties": {"format": "lance"}
     }'

2. Load the table. The response carries the resolved location, which Lance clients use to read and write the dataset.

   Load a Lance table

   curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables/orders" \
     -H "Accept: application/vnd.gravitino.v1+json" \
     -H "Authorization: Bearer $TOKEN"

To catalog a Lance dataset that already exists instead, see Register an existing dataset.

Supported operations

Operation	Supported	Notes
Create	Yes	Writes an empty Lance dataset at the table location.
Register	Yes	Records an existing dataset without writing to it. See Register an existing dataset.
List	Yes
Load	Yes	Rereads columns from the dataset as set by Schema refresh.
Alter	Partly	Drop a column, rename a column, and add an index. Every other change is rejected, including a column type change, renaming the table, changing its comment, and setting or removing a property.
Drop	Yes	Deletes the dataset as well unless the table is external.
Purge	Yes	Deletes the metadata and the dataset, external or not.

Partitioning, sort orders, and distributions are not supported.

Table properties

Property	Description	Default	Required
format	Table format. Must be lance.	(none)	Yes
location	Path of the Lance dataset, on S3, GCS, Azure storage, Aliyun OSS, or a local file system. Without it, the location comes from the schema or catalog; see Storage location.	(none)	No
external	When true, dropping the table removes only its metadata and leaves the dataset; purging still deletes both. When false, dropping deletes the dataset too.	false	No
lance.creation-mode	What happens when the table already exists. See Creation modes.	CREATE	No
lance.register	When true, the request records an existing dataset at location instead of writing a new one.	false	No
lance.storage.*	Lance storage options for this table, such as lance.storage.aws_access_key_id, overriding the catalog's lance.storage.* values. See the Lance storage options.	(none)	No

Creation modes

Mode	Behavior
CREATE	Creates the table and fails if it already exists.
EXIST_OK	Creates the table, or returns the existing one unchanged. Not allowed when registering.
OVERWRITE	Replaces an existing table. For a created table, the existing dataset is deleted first; for a registered table, only the old metadata is removed.

Data type mappings

Lance uses Apache Arrow for table schemas. The following table shows type mappings between Gravitino and Arrow:

Gravitino Type	Arrow Type
Struct	Struct
Map	Not supported by Lance
List	Array
Boolean	Boolean
Byte	Int8
Short	Int16
Integer	Int32
Long	Int64
Float	Float
Double	Double
String	Utf8
Binary	Binary
Decimal(p, s)	Decimal(p, s) (128-bit)
Date	Date
Timestamp/Timestamp(6)	TimestampType withoutZone
Timestamp(0)	TimestampType Second withoutZone
Timestamp(3)	TimestampType Millisecond withoutZone
Timestamp(9)	TimestampType Nanosecond withoutZone
Timestamp_tz/Timestamp_tz(6)	TimestampType Microsecond withUtc
Timestamp_tz(0)	TimestampType Second withUtc
Timestamp_tz(3)	TimestampType Millisecond withUtc
Timestamp_tz(9)	TimestampType Nanosecond withUtc
Time/Time(9)	Time Nanosecond
Null	Null
Fixed(n)	Fixed-Size Binary(n)
Interval_year	Not supported by Lance
Interval_day	Duration(Microsecond)
External(arrow_field_json_str)	Any Arrow Field

External types

For Arrow types not natively mapped in Gravitino, use the External(arrow_field_json_str) type, which accepts a JSON string representation of an Arrow Field.

The JSON must meet these rules:

• It follows the Apache Arrow Field specification
• name matches the column name exactly.
• nullable matches the column's nullability.
• children is empty for primitive types and holds the child fields for complex types such as Struct and List.

Examples:

Arrow Type	External Type Definition
Large Utf8	External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largeutf8\"},\"children\":[]}")
Large Binary	External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largebinary\"},\"children\":[]}")
Large List	External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largelist\"},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\",\"bitWidth\":32,\"isSigned\":true},\"children\":[]}]}")
Fixed-Size List	External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"fixedsizelist\",\"listSize\":10},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\",\"bitWidth\":32,\"isSigned\":true},\"children\":[]}]}")

Schema refresh

Gravitino stores a Lance table's columns in its own metadata store, but Lance writers can also change the dataset directly at its location. The catalog property lance.schema-refresh-mode decides when Gravitino rereads the columns from the dataset:

Mode	Behavior
DECLARED_AND_EMPTY	Default. Refreshes schema from the Lance dataset for two cases: (1) declared tables (lance.declared=true) whose schema has not yet been written to Gravitino; (2) tables whose Gravitino column list is empty, for example tables registered before their schema was captured.
VERSION_CHECK	Opens the Lance dataset on every loadTable, compares the dataset version with lance.version, and refreshes columns when the version has changed.

Use VERSION_CHECK only when tables may be modified directly through the Lance path outside Gravitino. It adds a dataset version check to every loadTable call.

If a Lance dataset has no columns, DECLARED_AND_EMPTY mode records the checked dataset version (lance.version) on the first loadTable call. Subsequent loads skip opening the dataset as long as the stored version is unchanged. Once columns are written to the dataset, the next VERSION_CHECK load or an explicit alterTable will detect the change and repair the schema.

Create a table

REST

Create a Lance table

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "lance_table",
    "comment": "Example Lance table",
    "columns": [
      {"name": "id", "type": "integer", "comment": "Primary identifier", "nullable": false}
    ],
    "properties": {
      "format": "lance",
      "location": "s3://{bucket}/lakehouse/sales/lance_table"
    }
  }'

Java

Create a Lance table

GravitinoClient gravitinoClient = GravitinoClient
    .builder("https://{gravitino_host}")
    .withMetalake("{metalake}")
    .withBasicAuth("{user}", "{password}")
    .build();
TableCatalog tableCatalog = gravitinoClient.loadCatalog("lakehouse_catalog").asTableCatalog();

Map<String, String> tableProperties = ImmutableMap.<String, String>builder()
    .put("format", "lance")
    .put("location", "s3://{bucket}/lakehouse/sales/lance_table")
    .build();

tableCatalog.createTable(
    NameIdentifier.of("sales", "lance_table"),
    new Column[] {
        Column.of("id", Types.IntegerType.get(), "Primary identifier", false, false, null)
    },
    "Example Lance table",
    tableProperties,
    null,  // partitions
    null,  // distribution
    null,  // sort orders
    null   // indexes
);

Load, list, alter, and drop work as for other tables; see Tables and views.

Register an existing dataset

Registering records a Lance dataset that already exists, without moving or copying it. Send no columns; Gravitino reads them from the dataset the first time the table is loaded. Set external to true as well, or dropping the table later deletes the dataset.

REST

Register an existing Lance dataset

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/sales/tables" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "existing_orders",
    "comment": "Registered Lance dataset",
    "columns": [],
    "properties": {
      "format": "lance",
      "lance.register": "true",
      "external": "true",
      "location": "s3://{bucket}/datasets/orders.lance"
    }
  }'

Java

Register an existing Lance dataset

Map<String, String> registerProperties = ImmutableMap.<String, String>builder()
    .put("format", "lance")
    .put("lance.register", "true")
    .put("external", "true")
    .put("location", "s3://{bucket}/datasets/orders.lance")
    .build();

tableCatalog.createTable(
    NameIdentifier.of("sales", "existing_orders"),
    new Column[] {},
    "Registered Lance dataset",
    registerProperties,
    null, null, null, null
);

Register many datasets

The script registers a list of existing datasets, one request each. Fill in the list with the schema, table name, and dataset location of each one; the schemas must already exist.

Register existing Lance datasets in bulk

datasets=(
  "sales orders s3://{bucket}/datasets/orders.lance"
  "sales customers s3://{bucket}/datasets/customers.lance"
  "inventory products s3://{bucket}/datasets/products.lance"
)

for entry in "${datasets[@]}"; do
  read -r schema table location <<< "$entry"
  curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/lakehouse_catalog/schemas/${schema}/tables" \
    -H "Accept: application/vnd.gravitino.v1+json" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $TOKEN" \
    -d "{
      \"name\": \"${table}\",
      \"columns\": [],
      \"properties\": {
        \"format\": \"lance\",
        \"lance.register\": \"true\",
        \"external\": \"true\",
        \"location\": \"${location}\"
      }
    }"
  echo "Registered ${schema}.${table}"
done

After registering, point Lance clients at the Lance REST service instead of the dataset paths, so reads and writes go through Gravitino.

S3-compatible storage

For Lance tables on MinIO or another S3-compatible store, set the storage options once on the catalog. Gravitino hands them to Lance clients with each table, so clients do not repeat them.

Create a catalog for Lance tables on MinIO

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "lance_catalog",
    "type": "RELATIONAL",
    "provider": "lakehouse-generic",
    "comment": "Lance tables on MinIO",
    "properties": {
      "location": "s3://{bucket}/lance",
      "lance.storage.endpoint": "https://{minio_host}:9000",
      "lance.storage.access_key_id": "{access_key_id}",
      "lance.storage.secret_access_key": "{secret_access_key}",
      "lance.storage.region": "us-east-1"
    }
  }'

For a MinIO endpoint on plain HTTP, add "lance.storage.allow_http": "true". A single table can override any of these with its own lance.storage.* properties.

Troubleshooting

Symptom	Cause and fix
'location' property is neither set in table properties...	None of the table, schema, or catalog sets location. Set it at one of the three levels.
Permission denied from storage	The storage credentials in lance.storage.* lack access to the location. Check them and the bucket policy.
A registered table has no columns	location does not point at a Lance dataset directory. Check the path, which usually ends in .lance.
Unsupported changes to lance table	The alter request changes something other than dropping or renaming a column or adding an index.

Related

• How Lance works
• Generic lakehouse catalog
• Lance REST service
• Lance integration