Skip to main content

Lance REST service

The Lance REST service lets Lance clients such as lance-spark, lance-ray, and pylance create, register, describe, alter, and drop Lance tables while Gravitino holds their metadata, so those tables are governed like any other Gravitino table. Lance is a columnar data format built for AI and machine learning workloads, with vector search, fast random access, and dataset versioning; see How Lance works. The service implements the Lance REST API specification; the Lance REST API reference lists the operations it serves, with examples.

Quick start​

  1. Set the metalake the service serves, as in Deployment.
  2. Publish the service on its own HTTPS host; see Client access.
  3. Run the example in the Lance REST API reference. It creates a namespace and a table, changes the table's columns, and drops it.

Architecture​

The Lance REST service runs inside the Datastrato Enterprise server and listens on its own port, 9101 by default. Lance clients such as lance-spark, lance-ray, and pylance call it for metadata. The service authenticates each caller the same way the Gravitino server does, then carries out the request by calling the Gravitino REST API under its own service identity, which Authentication to the Gravitino server configures. A two-level Lance namespace is a Gravitino catalog and schema, and a Lance table is a Gravitino table that records where its data lives. Clients then read and write Lance data directly at that location in object storage; the service touches the data files only when it creates a table or drops a table or namespace.

Lance clients call the Lance REST service inside the Datastrato Enterprise server for metadata; the service calls the Gravitino REST API as its service identity, and the metadata is stored in the metadata database as catalogs, schemas and tables, and read and write Lance data directly in object storage

Deployment​

The service runs in the server pod. Its server properties are below; set them through my-values.yaml and rerun helm upgrade --install.

Configuration PropertyDescriptionDefault ValueRequired
gravitino.auxService.namesAuxiliary services to run. Include lance-rest to enable Lance REST serviceiceberg-rest,lance-restYes
gravitino.lance-rest.classpathClasspath for Lance REST service, relative to Gravitino home directorylance-rest-server/libsYes
gravitino.lance-rest.httpPortPort number for Lance REST service9101No
gravitino.lance-rest.hostHostname for Lance REST service0.0.0.0No
gravitino.lance-rest.namespace-backendNamespace metadata backend (currently only gravitino is supported)gravitinoYes
gravitino.lance-rest.gravitino-uriGravitino server URI the service calls. Not set by the charthttp://localhost:8090No
gravitino.lance-rest.gravitino-metalakeGravitino metalake name (required when namespace-backend is gravitino)(none)Yes

The chart sets gravitino.auxService.names from the value of the same name, and each gravitino.lance-rest.* property above from the matching value under gravitino.lanceRest: classpath, host, httpPort, namespaceBackend and gravitinoMetalake. The chart does not render gravitino.lance-rest.gravitino-uri, so the service calls the server at its default address inside the pod. The chart's defaults match the table and already include lance-rest in gravitino.auxService.names, but gravitinoMetalake is empty by default and the chart does not check it. Until it is set, every Lance REST request fails, so the metalake is the one value you must set:

my-values.yaml
gravitino:
  lanceRest:
    gravitinoMetalake: "{metalake}"

Authentication to the Gravitino server​

The Lance REST service makes its own requests to the Gravitino server. Those requests must carry credentials, otherwise the Gravitino server rejects them and every Lance operation fails. The service reaches Gravitino with oauth2 or simple, of which Datastrato Enterprise accepts only oauth2. Configure it to match the Gravitino server:

Configuration PropertyDescriptionDefault ValueRequired
gravitino.lance-rest.gravitino-auth-typeAuth type used to reach the Gravitino server. Supported values: simple, oauth2simpleNo
gravitino.lance-rest.gravitino-oauth2.server-uriOAuth server URI(none)Yes, for oauth2
gravitino.lance-rest.gravitino-oauth2.credentialCredential used to request the OAuth token(none)Yes, for oauth2
gravitino.lance-rest.gravitino-oauth2.token-pathPath on the OAuth server used to request the token(none)Yes, for oauth2
gravitino.lance-rest.gravitino-oauth2.scopeScope of the requested OAuth token(none)Yes, for oauth2

These properties have no dedicated chart value. Set them under gravitino.additionalConfigItems as quoted strings; see Set Other Server Properties.

my-values.yaml
gravitino:
  additionalConfigItems:
    gravitino.lance-rest.gravitino-auth-type: "oauth2"
    gravitino.lance-rest.gravitino-oauth2.server-uri: "{oauth_server_uri}"
    gravitino.lance-rest.gravitino-oauth2.credential: "{client_id}:{client_secret}"
    gravitino.lance-rest.gravitino-oauth2.token-path: "{token_path}"
    gravitino.lance-rest.gravitino-oauth2.scope: "{scope}"

These properties control how the service authenticates to the Gravitino server. They do not change how callers authenticate to the Lance REST service itself.

Last updated on