Skip to main content

Files and unstructured data

Much of what AI and analytics teams work with never becomes a table: documents, images, audio, model checkpoints, raw extracts waiting to be loaded. A fileset catalog brings those files into the same catalog as your tables. Each fileset gives a directory in Amazon S3, Google Cloud Storage, Azure Data Lake Storage, Alibaba Cloud OSS or HDFS a name in Gravitino, and with the name come grants, tags, policies, an owner and audit. Code reads and writes the files by that name through GVFS and gets short-lived storage credentials from Gravitino, so it never holds a bucket path or a cloud key.

Documents, images and audio sit in object storage behind a fileset. An embedding job reads them through GVFS and writes vectors to a Lance table through the Lance REST service, and a search or RAG application queries the vectors. Gravitino governs both the fileset and the Lance table.

Common uses:

  • Source files for AI. Documents, images or audio that an embedding job turns into vectors in a Lance table, as in the diagram. The files and the vectors are governed in one place, by the same grants.
  • Landing zones. Raw extracts and logs that arrive before they are loaded into tables, with a name and an owner from the moment they land.
  • Training data and model files. Datasets and checkpoints shared across teams without handing out bucket paths or keys. Models themselves are registered in an ML model catalog.
  • Moving storage. Code uses the fileset's name, so moving the files to another bucket or cloud changes the fileset rather than every job that reads it.

Gravitino manages a fileset catalog itself rather than connecting an external system, so creating one needs no provider. Creating and managing filesets is covered in Filesets, and reading and writing the files in GVFS.

Quick start​

The example creates the fileset catalog files with its base location in S3, then the schema raw. Creating the schema creates its directory under the catalog's location, so it is the first step that tests the storage settings. Creating filesets in the schema is covered in Filesets.

  1. Start the catalog. Open Connect → Catalogs, click + Catalog, enter files as the Catalog name, and choose Fileset as the Catalog type.

  2. Enter the storage. Under Connection, choose S3 as the Storage and enter s3a://{bucket}/files as the Location. Enter https://s3.{region}.amazonaws.com as the S3 endpoint, and {access_key_id} and {secret_access_key} as the Access key ID and AWS secret key; the form requires all three for S3. AWS region and Path-style access are optional.

  3. Test and create it. Click Test connection, then Create catalog.

  4. Create the schema. In Explore, open files, click + Schema, and create raw.

Catalog properties​

These apply in addition to the common catalog properties.

Property NameDescriptionDefault Value
locationBase storage location, named unknown. Always a directory or path prefix, never a single file(none)
location-Prefix for named locations, as location-{name}={path}(none)
credential-providersCredential provider types, separated by commas(none)
config.resourcesConfiguration files to load, separated by commas, such as hdfs-site.xml,core-site.xml(none)
filesystem-conn-timeout-secsTimeout when obtaining a filesystem client, in seconds6
disable-filesystem-opsStops the server creating and removing directories when schemas and filesets are created and droppedfalse
fileset-cache-eviction-interval-msFileset cache eviction interval, where -1 never evicts3600000
fileset-cache-max-sizeMaximum filesets held in the cache, where -1 is unlimited200000
fs.path.config.{n}A logical location entry set to a base URI such as hdfs://cluster1/. Keys sharing the prefix are forwarded to that filesystem client(none)

The catalog finds file system providers on its own and picks one by the scheme of each location, so filesystem-providers is ignored. default-filesystem-provider, default builtin-local, only decides how a location written without a scheme is resolved.

Storage backends​

The server image includes the file system support for every storage system below, so object storage needs only its credential properties. HDFS and local file systems need none.

Storage systemURI schemeCredential providers
Amazon S3s3a://s3-token, s3-secret-key
Google Cloud Storagegs://gcs-token
Azure Data Lake Storageabfss://adls-token, azure-account-key
Alibaba Cloud OSSoss://oss-token, oss-secret-key
HDFS and localhdfs://, file://None

The GVFS client takes the same property names as the catalog, so a client reading an S3 fileset sets s3-endpoint, s3-access-key-id, and s3-secret-access-key alongside its base GVFS configuration. Setting credential-providers on the catalog removes that requirement, since Gravitino then issues short-lived credentials per request and the client holds no cloud keys at all. See credential vending.

Amazon S3​

Property NameDescriptionRequired
s3-endpointEndpoint of the S3 serviceYes
s3-access-key-idAccess keyYes
s3-secret-access-keySecret keyYes

S3-compatible storage such as MinIO uses the same properties with its own endpoint.

Create an S3 fileset catalog
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "{catalog_name}",
    "type": "FILESET",
    "comment": "",
    "properties": {
      "location": "s3a://{bucket}/{prefix}",
      "s3-endpoint": "{s3_endpoint}",
      "s3-access-key-id": "{access_key_id}",
      "s3-secret-access-key": "{secret_access_key}"
    }
  }'

Google Cloud Storage​

Property NameDescriptionRequired
gcs-service-account-filePath to the service account JSON fileYes

The path is read where it is used: in the server pod for the catalog, and on the client machine for a client not using vended credentials. Mount the key into the server pod from a Secret, as described in Google Cloud credentials.

Azure Data Lake Storage​

Property NameDescriptionRequired
azure-storage-account-nameStorage account nameYes
azure-storage-account-keyStorage account keyYes

Alibaba Cloud OSS​

Property NameDescriptionRequired
oss-endpointEndpoint of the OSS serviceYes
oss-access-key-idAccess keyYes
oss-secret-access-keySecret keyYes

Multiple storage systems​

One catalog can carry the properties for several storage systems at once, and Gravitino selects among them by the URI scheme of the object being accessed.

HDFS and Kerberos​

A secured HDFS cluster needs these on the catalog, and they can be narrowed on a schema or fileset.

Property NameDescriptionDefault Value
authentication.typesimple or kerberossimple
authentication.impersonation-enableWhether the catalog impersonates the calling userfalse
authentication.kerberos.principalKerberos principal, required when the type is kerberos(none)
authentication.kerberos.keytab-uriURI of the keytab, required when the type is kerberos(none)
authentication.kerberos.check-interval-secCredential check interval60
authentication.kerberos.keytab-fetch-timeout-secTimeout when retrieving the keytab60

The HDFS client itself is configured through the files named in config.resources, which are paths in the server pod; mount them there with gravitino.extraVolumes and gravitino.extraVolumeMounts, as described in Customize the Helm chart. In those files Gravitino recognizes three additional keys: hadoop.security.authentication.kerberos.principal, hadoop.security.authentication.kerberos.keytab, and hadoop.security.authentication.kerberos.krb5.conf.

Schema properties​

Schemas inherit every catalog property and can override these.

Property NameDescriptionDefault Value
locationBase storage location for the schema, named unknown(none)
location-Prefix for named locations(none)
credential-providersCredential provider types, separated by commas(none)
config.resourcesConfiguration files to load(none)
authentication.typesimple or kerberosCatalog value
authentication.impersonation-enableWhether to impersonate the calling userCatalog value
authentication.kerberos.principalKerberos principal for this schemaCatalog value
authentication.kerberos.keytab-uriKeytab URI for this schemaCatalog value

Creating or dropping a schema creates or removes the matching directories, except when disable-filesystem-ops is true or the location contains placeholders.

Fileset properties​

Filesets inherit every schema property, including those the schema inherited from the catalog.

Property NameDescriptionDefault ValueImmutable
locationStorage location for the fileset, named unknown(none)No
default-location-nameWhich location GVFS uses when none is named. Required with severalThe only location, when there is oneYes
placeholder-Values substituted into placeholders in the location(none)Yes
credential-providersCredential provider types, separated by commas(none)No
config.resourcesConfiguration files to load(none)No
authentication.typesimple or kerberosSchema valueNo
authentication.impersonation-enableWhether to impersonate the calling userSchema valueYes
authentication.kerberos.principalKerberos principal for this filesetSchema valueNo
authentication.kerberos.keytab-uriKeytab URI for this filesetSchema valueNo

Three placeholders are supplied by Gravitino and cannot be set: placeholder-catalog, placeholder-schema, and placeholder-fileset, which resolve to the names of the objects themselves.

Property inheritance​

Authentication and credential settings resolve from the nearest level outward, so a fileset value beats a schema value, which beats a catalog value. A catalog can therefore set a default that individual filesets override without repeating the rest of the configuration.

Implement a custom filesystem provider​

A storage system without built-in support can be added by implementing FileSystemProvider from gravitino-hadoop-common:

FileSystemProvider methods to implement
// Build a FileSystem from the properties set when the catalog was created.
FileSystem getFileSystem(@Nonnull Path path, @Nonnull Map<String, String> config)
    throws IOException;

// URI scheme, such as 'file', 'hdfs', 's3a', 'gs', or 'oss'.
String scheme();

// Provider name, such as 'builtin-local', 'builtin-hdfs', 's3', 'gcs', or 'oss'.
String name();

The provider is discovered through Java SPI, so the jar needs a META-INF/services/org.apache.gravitino.catalog.hadoop.fs.FileSystemProvider file naming the implementing class. The jar has to be in /opt/gravitino/catalogs/fileset/libs/ in the server container when the server starts. Mount it there from a volume with a subPath, so the jars already in that directory stay, using gravitino.extraVolumes and gravitino.extraVolumeMounts.

Last updated on