Skip to main content

Filesets

A fileset is a named pointer to files. Where a table describes rows and columns, a fileset describes a location, so unstructured and semi-structured data gets the same treatment as everything else in the catalog: a name, a place in the hierarchy, tags, policies, and an owner.

The point is indirection. Code that reads a fileset by name does not carry a bucket path, so moving data between clusters or storage systems changes the fileset rather than every job that reads it.

That indirection is delivered by the Gravitino Virtual File System, or GVFS. GVFS is a filesystem implementation that resolves a fileset name to its storage location and then reads and writes through to the underlying system, whether that is HDFS, S3, GCS, ADLS, or OSS. Paths take the form gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}, so a Spark job, a pandas script, or a Hadoop shell command reads a catalog name rather than a bucket URL.

GVFS comes in two implementations. The Java one implements the Hadoop Compatible File System interface, so anything that already speaks HDFS paths works, and it requires Hadoop 3.3.1 or later. The Python one is built on fsspec, so it works with pandas, PyArrow, and anything else in that ecosystem.

Credentials are the other half of it. A caller reaching storage through GVFS can be given short-lived credentials vended by Gravitino rather than holding long-lived cloud keys of its own, which is what makes the indirection a governance boundary rather than only a convenience. See Credential vending.

Filesets live in a schema inside a fileset catalog. Gravitino manages that catalog itself rather than federating an external one, so no provider is needed when creating it.

Quick start​

The example creates the fileset catalog files with a base location, the schema raw in it, and a managed fileset events whose directory Gravitino creates beneath that location. It then lists the files in the fileset.

  1. Create the catalog. Open Connect → Catalogs, click + Catalog, enter files as the Catalog name, and choose Fileset as the Catalog type. Under Connection, choose HDFS as the Storage and enter hdfs://namenode:8020/data/files as the Location. The other storage choices are S3, ADLS, GCS, OSS, and Local, each with its own credential fields. Click Test connection, then Create catalog.

  2. Create the schema. In Explore, open files, click + Schema, and create raw.

  3. Create the fileset. Open raw, click + Fileset, and enter events as the Name. Keep Managed as the Type and leave Storage locations empty, so the fileset takes its place under the schema. Click Create fileset. For an external fileset, choose External and enter the existing path under Storage locations, with its scheme and an optional name; + Location adds another.

  4. List its files. Open events. The Files tab shows the fileset's storage location and lists the files and directories in it with their size and modification time. The Overview tab shows the type, storage locations, creator, and owner, and edits the properties.

To tag filesets or attach policies to them, select their rows in the schema and click + Tags or + Policies. The header of a fileset's page shows its grants, tags, and policies, with a control for each.

Engines and scripts read and write the files by name through GVFS, at gvfs://fileset/files/raw/events. See How to use GVFS.

The fileset model​

Managed and external​

A managed fileset belongs to Gravitino. Creating it creates the directory, and deleting it deletes the data.

An external fileset points at a location that already exists and stays under someone else's control. Deleting it removes the Gravitino record and leaves the files alone.

The distinction matters most at deletion time, which makes it the most important choice to get right when creating a fileset.

Storage locations​

A fileset has at least one storage location and can have several, each with a name. The default is chosen by the default-location-name property, and a location supplied without a name is recorded as unknown.

Several locations on one fileset is how the same logical dataset is described across clusters or regions, with readers selecting the one they should use rather than each carrying its own path.

Locations can also be inherited: a catalog or schema with a location property supplies a base, and a fileset created beneath it can take its placement from that rather than spelling out a full path.

A location can be a template rather than a fixed path. Placeholders are filled in when the fileset is created. {{catalog}}, {{schema}}, and {{fileset}} take the names of the fileset and its parents, and any other placeholder, such as {{project}}, takes its value from the fileset's placeholder- property of the same name. One catalog-level template then generates a consistent directory layout for every fileset created beneath it, rather than each one being spelled out by hand. With a catalog location of s3a://example-bucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}, a fileset created in catalog landing and schema raw with placeholder-project set to risk and placeholder-user set to jdoe resolves to s3a://example-bucket/landing/raw/workspace_risk/jdoe.

Storage locations, default-location-name, and the placeholder- properties are fixed when the fileset is created and cannot be changed afterward.

Properties​

Properties are free-form, with default-location-name and the location-{name} keys reserved for placement. Everything else is yours, and travels with the fileset.

Permissions​

Creating a fileset requires CREATE_FILESET on its schema or above. Reading a fileset's files requires READ_FILESET, and writing them requires WRITE_FILESET.

Granting at a wider scope covers everything beneath it. Dropping a fileset is reserved for the metalake owner and the object owner.

See Privileges for a comprehensive list of privilege grants.

Use the API​

Filesets can be created, listed, altered, and dropped over REST and through the Java and Python clients. The REST reference documents each call, with the client code under the Java and Python tabs.

Last updated on