Manage Fileset Metadata

Introduction

This page covers the Gravitino API for filesets. For what a fileset is, the difference between managed and external, how storage locations and placeholders work, and how GVFS reads them, see Filesets. For creating the catalog and schema a fileset lives in, see Manage Catalogs and Schemas.

Fileset Operations

Create a Fileset

A fileset needs a name, a type, and a storage location. A MANAGED fileset is created and deleted with its data; an EXTERNAL one points at a location that already exists.

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "raw_events",
  "comment": "Raw event drops",
  "type": "MANAGED",
  "storageLocation": "s3a://example-bucket/landing/raw_events",
  "properties": {"retention": "30d"}
}' http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets

Java

Catalog catalog = client.loadCatalog("landing");
FilesetCatalog filesets = catalog.asFilesetCatalog();

Fileset fileset = filesets.createFileset(
    NameIdentifier.of("raw", "raw_events"),
    "Raw event drops",
    Fileset.Type.MANAGED,
    "s3a://example-bucket/landing/raw_events",
    ImmutableMap.of("retention", "30d"));

Python

catalog = client.load_catalog("landing")
filesets = catalog.as_fileset_catalog()

fileset = filesets.create_fileset(
    ident=NameIdentifier.of("raw", "raw_events"),
    comment="Raw event drops",
    fileset_type=Fileset.Type.MANAGED,
    storage_location="s3a://example-bucket/landing/raw_events",
    properties={"retention": "30d"})

Create a Fileset With Several Locations

storageLocations takes a map of location name to path. The name is what a reader selects, and default-location-name picks the one used when none is named.

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "raw_events",
  "comment": "Raw event drops, two regions",
  "type": "MANAGED",
  "storageLocations": {
    "us": "s3a://example-us/landing/raw_events",
    "eu": "s3a://example-eu/landing/raw_events"
  },
  "properties": {"default-location-name": "us"}
}' http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets

Create a Fileset From a Location Template

A catalog or schema can carry a location template, and a fileset created beneath it fills the placeholders from its own placeholder- properties.

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "workspace",
  "type": "MANAGED",
  "properties": {
    "placeholder-project": "risk",
    "placeholder-user": "mhoerth"
  }
}' http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets

With a catalog location of s3a://example-bucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}, that request resolves to s3a://example-bucket/landing/raw/workspace_risk/mhoerth.

Load a Fileset

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets/raw_events

Java

Fileset fileset = filesets.loadFileset(NameIdentifier.of("raw", "raw_events"));

Python

fileset = filesets.load_fileset(NameIdentifier.of("raw", "raw_events"))

Alter a Fileset

Changes are applied as a list in one request. Storage locations cannot be changed after creation.

Change	JSON	Java
Rename	{"@type":"rename","newName":"fileset_renamed"}	FilesetChange.rename("fileset_renamed")
Update the comment	{"@type":"updateComment","newComment":"new_comment"}	FilesetChange.updateComment("new_comment")
Set a property	{"@type":"setProperty","property":"key1","value":"value1"}	FilesetChange.setProperty("key1", "value1")
Remove a property	{"@type":"removeProperty","property":"key1"}	FilesetChange.removeProperty("key1")

REST

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "updates": [
    {"@type": "setProperty", "property": "retention", "value": "90d"}
  ]
}' http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets/raw_events

Java

Fileset fileset = filesets.alterFileset(
    NameIdentifier.of("raw", "raw_events"),
    FilesetChange.setProperty("retention", "90d"));

Drop a Fileset

Dropping a MANAGED fileset deletes its files. Dropping an EXTERNAL one removes only the Gravitino record.

REST

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets/raw_events

Java

boolean dropped = filesets.dropFileset(NameIdentifier.of("raw", "raw_events"));

Python

dropped = filesets.drop_fileset(NameIdentifier.of("raw", "raw_events"))

List Filesets

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/landing/schemas/raw/filesets

Java

NameIdentifier[] identifiers = filesets.listFilesets(Namespace.of("raw"));

Python

identifiers = filesets.list_filesets(Namespace.of("raw"))

Reading and Writing Files

Fileset metadata operations do not move data. Reading and writing the files themselves goes through GVFS. See How to use GVFS.