How Lance works

Lance is an open columnar table format built for AI data. A Lance table holds ordinary columns next to embedding vectors, long text and images, and serves both large scans and fast lookups of individual rows. Its most common job is the searchable store behind semantic search and retrieval augmented generation (RAG): each row is a passage from a document, the passage's embedding vector, and a pointer to the file it came from.

In Datastrato Enterprise, a Lance table belongs to a generic lakehouse catalog. Gravitino records the table's name, columns and location, and Lance clients such as pylance, lance-spark and lance-ray reach it through the Lance REST service. The page covers how the format stores a table, how vector search works, and how to go from a folder of files to a table an application can search.

Quick start

The example writes a three-row table with vectors through the Lance REST service, then finds the two rows nearest to a query vector. It assumes the catalog lance_catalog and the schema docs exist, as created in the Lance REST API example, and an access token in TOKEN, obtained as in the Lance engines Quick start.

1. Install the Lance Python packages.

   Install pylance and the namespace client

   pip install pylance lance-namespace

2. Write a table and search it.

   quickstart.py

   import os
   import lance
   import lance_namespace as ln
   import pyarrow as pa
   
   namespace = ln.connect("rest", {
       "uri": "https://{lance_host}/lance",
       "header.Authorization": f"Bearer {os.environ['TOKEN']}",
   })
   table_id = ["lance_catalog", "docs", "quickstart"]
   
   data = pa.table({
       "text": ["the cat sat", "dogs bark", "a kitten naps"],
       "vector": pa.array([[0.9, 0.1, 0.0], [0.0, 0.2, 0.9], [0.8, 0.2, 0.1]],
                          pa.list_(pa.float32(), 3)),
   })
   lance.write_dataset(data, namespace_client=namespace, table_id=table_id)
   
   ds = lance.dataset(namespace_client=namespace, table_id=table_id)
   print(ds.to_table(nearest={"column": "vector", "q": [0.85, 0.15, 0.05], "k": 2}).to_pandas())

   The two nearest rows are the cat sat and a kitten naps, each with a _distance column. Real vectors come from an embedding model and have hundreds of dimensions, as in From files to a searchable table.

Lance compared with Parquet

Iceberg and Delta tables store their data in Parquet, which is built for scans: reading a few columns across many rows. AI workloads also need the opposite, fetching a few whole rows by position, and searching by similarity rather than by value. Lance is built for both.

Need	Parquet tables	Lance tables
Scan a few columns	Fast	Fast
Fetch individual rows	Slow: a whole page of each column is read and decoded per lookup	Fast: rows are read directly by position
Similarity search on vectors	No index; every vector is compared	Vector indexes stored with the table
Fill a new column for existing rows	Rewrites the data files	Writes the new column only; existing files are untouched
Large values such as images	Inflate pages and slow scans of other columns	Stored as blobs, read only when asked for
History	Snapshots, with time travel	Versions, with time travel

Use Iceberg for analytics tables that engines such as Trino and Spark scan and join. Use Lance for data that applications retrieve, search, or feed to model training. One Datastrato Enterprise deployment catalogs both.

Table structure

A Lance table is a directory in object storage, called a dataset. Gravitino records the table's name and where the dataset lives. Inside the dataset, a manifest describes each version of the table: its schema, its fragments, and its indexes. A fragment is a group of rows, stored in one or more data files, with an optional file recording which of its rows are deleted.

Layout of a Lance dataset

contract_chunks.lance/
  _versions/       one manifest per version
  data/            data files, *.lance
  _deletions/      deletion files, one per fragment that has deleted rows
  _indices/        one directory per index
  _transactions/   one file per commit, used to resolve concurrent writes
  _refs/tags/      named tags on versions

Clients read and write the dataset directly. The Lance REST service tells them where the table is and which storage options to use; it does not carry the data.

Fragments

Each write appends new fragments rather than changing existing files, which is what makes adding data, adding columns and deleting rows cheap.

Change	What Lance writes
Insert	One or more new fragments
Add a column	One data file per existing fragment, holding only the new column
Delete	A deletion file for each fragment that held a deleted row
Update	A deletion file for the old row, and the new row in a new fragment
Compaction	Fewer, larger fragments, with deleted rows removed

Adding a column is how a Lance table grows with a pipeline. A table of chunks can gain a second embedding from a newer model, or a summary column, without rewriting the text and vectors it already holds.

Versions and time travel

Every write commits a new version of the table, described by a new manifest. Older manifests stay, so any version that has not been cleaned up can be opened and read as it was.

Open an earlier version

ds = lance.dataset(namespace_client=namespace, table_id=table_id, version=3)

A tag names a version, so it can be opened by name and is protected from cleanup:

Tag a version

ds.tags.create("before_reembed", ds.version)
old = lance.dataset(namespace_client=namespace, table_id=table_id, version="before_reembed")

A writer commits by creating the manifest for the next version number, and storage lets only one writer create it. When two writers race, the second finds the version taken, reads what the first committed, and retries on top of it when the two changes are compatible: two appends, for example, or deletes of different rows. Changes that cannot be combined fail with an error instead of overwriting each other.

Embeddings and vector search

An embedding model turns a piece of text, or an image, into a vector: a fixed-length list of numbers that places it in a space where similar meaning is close together. A search embeds the question with the same model and returns the rows whose vectors are nearest, so it finds passages that answer the question even when they share none of its words.

A Lance table stores each vector in a fixed-size list column, such as fixed_size_list<float>[384] for a model with 384 dimensions. Every vector in a column must come from the same model, because vectors from different models cannot be compared.

Vector indexes

Without an index, a search compares the query with every vector in the table, which is fine for tens of thousands of rows and slow for many millions. A vector index groups the vectors so a search reads only a small part of them.

Create a vector index

ds.create_index("vector", index_type="IVF_PQ", metric="cosine",
                num_partitions=256, num_sub_vectors=48)

Index type	How it works
IVF_PQ	Partitions the vectors, and compresses each one to a short code; small and fast, approximate
IVF_HNSW_SQ	Partitions the vectors, and links each partition into a graph; higher recall, larger
IVF_FLAT	Partitions the vectors and keeps them uncompressed; exact within the partitions searched

metric is the distance the index measures: cosine for most text embedding models, or l2 or dot when the model's documentation says so. More partitions mean each search reads a smaller share of the table. For IVF_PQ, num_sub_vectors must divide the vector's dimension evenly; 48 divides 384.

A search sets how many partitions to read and how many candidates to re-rank with exact distances. A filter can narrow the rows first:

Search with a filter

hits = ds.to_table(
    nearest={"column": "vector", "q": query_vector, "k": 5, "nprobes": 20, "refine_factor": 10},
    filter="source = 'contracts/acme-msa.pdf'",
    prefilter=True,
    columns=["source", "text"],
)

Lance also builds a full-text index, ds.create_scalar_index("text", "INVERTED"), for keyword search on the same table, and scalar indexes on ordinary columns to speed up filters.

From files to a searchable table

Getting unstructured files into Lance is a pipeline in your own code. It reads each file, pulls out the text, splits it into passages, embeds each passage, and writes the passages and vectors as rows.

The example turns a folder of PDF contracts into the table lance_catalog.docs.contract_chunks. It uses pypdf to extract text and the open all-MiniLM-L6-v2 model from sentence-transformers, which runs locally and returns 384-dimensional vectors; any parser and any embedding model fit the same steps.

Install the pipeline's packages

pip install pylance lance-namespace pypdf sentence-transformers

1. Extract and chunk the text. A chunk of about 800 characters holds a paragraph or two, small enough to be about one thing and large enough to make sense alone. The 100-character overlap keeps a sentence cut at a boundary whole in one of the two chunks.

   pipeline.py

   import glob
   import os
   import lance
   import lance_namespace as ln
   import pyarrow as pa
   from pypdf import PdfReader
   from sentence_transformers import SentenceTransformer
   
   def chunks(text, size=800, overlap=100):
       for start in range(0, max(len(text) - overlap, 1), size - overlap):
           yield text[start:start + size]
   
   rows = []
   for path in sorted(glob.glob("contracts/*.pdf")):
       text = "\n".join(page.extract_text() or "" for page in PdfReader(path).pages)
       for i, chunk in enumerate(chunks(text)):
           rows.append({"id": f"{path}#{i}", "source": path, "chunk": i, "text": chunk})

2. Embed every chunk. The model runs on the chunks in batches. Normalized vectors suit the cosine metric.

   pipeline.py, continued

   model = SentenceTransformer("all-MiniLM-L6-v2")
   vectors = model.encode([r["text"] for r in rows], normalize_embeddings=True)

3. Build the table. Each row keeps the passage, where it came from, and its vector, so a search result can be shown with its source.

   pipeline.py, continued

   table = pa.table({
       "id": [r["id"] for r in rows],
       "source": [r["source"] for r in rows],
       "chunk": [r["chunk"] for r in rows],
       "text": [r["text"] for r in rows],
       "vector": pa.FixedSizeListArray.from_arrays(pa.array(vectors.ravel(), pa.float32()), 384),
   })

4. Write it through the Lance REST service. Gravitino records the new table under lance_catalog.docs, and the data lands under the catalog's location.

   pipeline.py, continued

   namespace = ln.connect("rest", {
       "uri": "https://{lance_host}/lance",
       "header.Authorization": f"Bearer {os.environ['TOKEN']}",
   })
   table_id = ["lance_catalog", "docs", "contract_chunks"]
   ds = lance.write_dataset(table, namespace_client=namespace, table_id=table_id)

5. Index and search. The question is embedded with the same model as the chunks.

   pipeline.py, continued

   ds.create_index("vector", index_type="IVF_PQ", metric="cosine",
                   num_partitions=16, num_sub_vectors=48)
   
   question = "How much notice do we need to give to end the contract?"
   query_vector = model.encode([question], normalize_embeddings=True)[0]
   hits = ds.to_table(nearest={"column": "vector", "q": query_vector, "k": 3},
                      columns=["source", "text"])

When new files arrive, run steps 1 to 3 on them and append. When a file changes, delete its old chunks and append the new ones, so the table never holds two versions of the same passage:

Refresh one file's chunks

ds = lance.dataset(namespace_client=namespace, table_id=table_id)
ds.delete("source = 'contracts/acme-msa.pdf'")
lance.write_dataset(new_rows, namespace_client=namespace, table_id=table_id, mode="append")

To keep the source files governed as well, put them in a fileset and read them through GVFS in step 1, so the pipeline reads files by their Gravitino name rather than by bucket path.

Answer questions from the table

A RAG application answers a question in two steps: retrieve the passages most likely to hold the answer, then hand them to a language model to write the answer from.

Retrieve passages and build the prompt

hits = ds.to_table(nearest={"column": "vector", "q": query_vector, "k": 5},
                   columns=["source", "text"]).to_pylist()
context = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
prompt = f"Answer from these passages and cite the source.\n\n{context}\n\nQuestion: {question}"

The prompt goes to whichever language model the application uses. Because the answer is written from retrieved passages, it can cite the files they came from, and a better answer comes from better chunks and a better embedding model rather than from a different language model.

Maintenance

Appends add small fragments, deletes add deletion files, and every write keeps the previous version. Three operations keep a table fast and its storage in check:

Operation	What it does	Python
Compact files	Merges small fragments into larger ones and removes deleted rows	ds.optimize.compact_files()
Optimize indices	Adds rows written since the index was built, which are otherwise searched without it	ds.optimize.optimize_indices()
Clean up	Deletes versions older than a threshold and the files only they used, which ends time travel to them	ds.cleanup_old_versions(older_than=...)

Cleanup refuses to remove a tagged version and fails with an error naming the tag; delete the tag first, or pass error_if_tagged_old_versions=False to keep tagged versions and remove the rest.

Related

• Lance REST service
• Lance tables
• Generic lakehouse catalog
• Lance engines
• Fileset catalog
• How Iceberg works
• Lance format specification