Skip to main content

Gravitino Virtual File System

The Gravitino Virtual File System (GVFS) reads and writes the files in a fileset through a virtual path, so code never names the storage behind it. The server resolves each path to the fileset's real location in HDFS or object storage and applies the fileset's permissions on the way.

GVFS comes in two forms with the same paths and the same behavior:

  • Java, a Hadoop Compatible File System. Anything that reads Hadoop paths uses it: the Hadoop shell, Java code, Spark, and TensorFlow.
  • Python, an fsspec file system. Anything that accepts an fsspec file system uses it: Pandas, PyArrow, Ray, and LlamaIndex.

Each section below has a Java and a Python tab. Choosing one switches every section on the page.

Quick start​

The example lists a fileset, writes a file into it, and reads the file back. It assumes the fileset sales.raw.events exists in the metalake example, and signs in with an OAuth client credential; see Authentication for the identity provider side.

  1. Add the GVFS jar to Hadoop. Datastrato Enterprise 1.3.0 is built on Apache Gravitino 1.3.1, so use version 1.3.1 of the jar.

    Download the GVFS runtime jar
    cd ${HADOOP_HOME}/share/hadoop/common/lib/
    curl -sSfLO https://repo1.maven.org/maven2/org/apache/gravitino/gravitino-filesystem-hadoop3-runtime/1.3.1/gravitino-filesystem-hadoop3-runtime-1.3.1.jar

  2. Point Hadoop at the server. Add these properties to ${HADOOP_HOME}/etc/hadoop/core-site.xml.

    core-site.xml
    <property>
      <name>fs.AbstractFileSystem.gvfs.impl</name>
      <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
    </property>
    <property>
      <name>fs.gvfs.impl</name>
      <value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
    </property>
    <property>
      <name>fs.gravitino.server.uri</name>
      <value>https://{gravitino_host}</value>
    </property>
    <property>
      <name>fs.gravitino.client.metalake</name>
      <value>example</value>
    </property>
    <property>
      <name>fs.gravitino.client.authType</name>
      <value>oauth2</value>
    </property>
    <property>
      <name>fs.gravitino.client.oauth2.serverUri</name>
      <value>https://{idp_host}</value>
    </property>
    <property>
      <name>fs.gravitino.client.oauth2.path</name>
      <value>{token_path}</value>
    </property>
    <property>
      <name>fs.gravitino.client.oauth2.credential</name>
      <value>{client_id}:{client_secret}</value>
    </property>
    <property>
      <name>fs.gravitino.client.oauth2.scope</name>
      <value>{scope}</value>
    </property>

    {token_path} is the token endpoint's path without its leading slash, for example realms/{realm}/protocol/openid-connect/token for Keycloak.

  3. Use the fileset.

    List, write, and read a fileset
    hadoop fs -ls gvfs://fileset/sales/raw/events/
    echo "hello world" > hello.txt
    hadoop fs -put hello.txt gvfs://fileset/sales/raw/events/hello.txt
    hadoop fs -cat gvfs://fileset/sales/raw/events/hello.txt

Paths​

A GVFS path names the catalog, schema, and fileset, followed by any path inside the fileset.

GVFS path
gvfs://fileset/{catalog}/{schema}/{fileset}/{sub_path}

gvfs is the scheme and fileset is a fixed root. Python also accepts the path without the scheme, as fileset/{catalog}/{schema}/{fileset}/{sub_path}.

Choose a location​

A fileset can have several storage locations. GVFS picks one in this order:

  1. The location named in the client configuration: fs.gravitino.current.location.name in Java, current_location_name in Python.
  2. The location named in the environment variable CURRENT_LOCATION_NAME. The variable's name can be changed with fs.gravitino.current.location.name.env.var in Java or current_location_name_env_var in Python.
  3. The fileset's default-location-name property.

Configuration​

GVFS for Java needs Hadoop 3.3.1 or later. Properties go in core-site.xml, on a Hadoop Configuration in code, or as spark.hadoop. options in Spark.

Configuration itemDescriptionDefault valueRequired
fs.AbstractFileSystem.gvfs.implThe Gravitino Virtual File System abstract class, set it to org.apache.gravitino.filesystem.hadoop.Gvfs.(none)Yes
fs.gvfs.implThe Gravitino Virtual File System implementation class, set it to org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem.(none)Yes
fs.gvfs.impl.disable.cacheDisable the Gravitino Virtual File System cache in the Hadoop environment. If you need to proxy multi-user operations, set this value to true and create a separate File System for each user.falseNo
fs.gravitino.server.uriThe Gravitino server URI which GVFS needs to load the fileset metadata.(none)Yes
fs.gravitino.client.metalakeThe metalake to which the fileset belongs.(none)Yes
fs.gravitino.client.authTypeThe auth type the Gravitino client uses with the Gravitino Virtual File System: oauth2. See Authentication.simpleYes
fs.gravitino.client.oauth2.serverUriThe auth server URI for the Gravitino client when using oauth2 auth type with the Gravitino Virtual File System.(none)Yes if you use oauth2 auth type
fs.gravitino.client.oauth2.credentialThe auth credential for the Gravitino client when using oauth2 auth type in the Gravitino Virtual File System.(none)Yes if you use oauth2 auth type
fs.gravitino.client.oauth2.pathThe auth server path for the Gravitino client when using oauth2 auth type with the Gravitino Virtual File System. Remove the first slash / from the path, for example oauth/token.(none)Yes if you use oauth2 auth type
fs.gravitino.client.oauth2.scopeThe auth scope for the Gravitino client when using oauth2 auth type with the Gravitino Virtual File System.(none)Yes if you use oauth2 auth type
fs.gravitino.fileset.cache.maxCapacityThe cache capacity of the Gravitino Virtual File System.20No
fs.gravitino.fileset.cache.evictionMillsAfterAccessThe value of time that the cache expires after accessing in the Gravitino Virtual File System. The value is in milliseconds.3600000No
fs.gravitino.fileset.cache.closeOnEvictionWhether to close the underlying FileSystem immediately on cache eviction. Default is false: close is deferred to GVFS shutdown to avoid breaking long-lived streams. Set to true to close immediately on eviction.falseNo
fs.gravitino.current.location.nameThe location of the fileset to use; see Choose a location.the value of fileset property default-location-nameNo
fs.gravitino.current.location.name.env.varThe environment variable name to get the current location name.CURRENT_LOCATION_NAMENo
fs.gravitino.operations.classThe operations class to provide the FS operations for the Gravitino Virtual File System. Users can extends BaseGVFSOperations to implement their own operations and configure the class name in this conf to use custom FS operations.org.apache.gravitino.filesystem.hadoop.DefaultGVFSOperationsNo
fs.gravitino.hook.classThe hook class to inject into the Gravitino Virtual File System. Users can implement their own GravitinoVirtualFileSystemHook and configure the class name in this conf to inject custom code.org.apache.gravitino.filesystem.hadoop.NoOpHookNo
fs.gravitino.client.request.header.The configuration key prefix for the Gravitino client request header. You can set the request header for the Gravitino client.(none)No
fs.gravitino.enableCredentialVendingWhether to enable credential vending for the Gravitino Virtual File System.falseNo
fs.gravitino.client.The configuration key prefix for the Gravitino client config.(none)No
fs.gravitino.filesetMetadataCache.enableWhether to cache the fileset, fileset schema or fileset catalog metadata in the Gravitino Virtual File System. Note that this cache causes a side effect: if you modify the fileset or fileset catalog metadata, the client cannot see the latest changes.falseNo
fs.gravitino.autoCreateLocationThe configuration key for whether to enable auto-creation of fileset location when the server-side filesystem ops are disabled and the location does not exist.trueNo
fs.path.config.{name}Defines a logical location entry. Set fs.path.config.{name} to the real base URI (for example, hdfs://cluster1/). Any key that starts with the same prefix (such as fs.path.config.{name}.config.resource) is treated as a location-scoped property and will be forwarded to the underlying filesystem client. Note: location names must not contain (., _).(none)No

Properties prefixed with fs.gravitino.client. configure the underlying Gravitino client, with the fs. removed: fs.gravitino.client.socketTimeoutMs sets gravitino.client.socketTimeoutMs. An unknown client property raises an exception; see Java client configurations for the full list.

Filesets spread across clusters. When a fileset catalog spans several HDFS clusters, give each cluster's base path its own client settings with fs.path.config.{name}.

Client properties per base path
fs.path.config.cluster1 = hdfs://cluster1/
fs.path.config.cluster1.config.resource = /etc/core-site.xml,/etc/hdfs-site.xml

fs.path.config.cluster2 = hdfs://cluster2/
fs.path.config.cluster2.config.resource = /etc/fs2/core-site.xml,/etc/fs2/hdfs-site.xml

Object storage. Filesets on S3, GCS, OSS, or Azure need storage properties as well; see Amazon S3, Google Cloud Storage, Alibaba Cloud OSS, and Azure Data Lake Storage.

Custom file systems. A fileset on a custom file system provider takes whatever properties that provider's getFileSystem reads, set the same way as the properties above.

Authentication​

Datastrato Enterprise does not accept the simple auth type, which is the client default, so every client sets an auth type. Configure the server side first; see Authentication.

GVFS for Java signs in with OAuth.

Configure GVFS with OAuth
Configuration conf = new Configuration();
conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs");
conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
conf.set("fs.gravitino.server.uri", "https://{gravitino_host}");
conf.set("fs.gravitino.client.metalake", "example");
conf.set("fs.gravitino.client.authType", "oauth2");
conf.set("fs.gravitino.client.oauth2.serverUri", "https://{idp_host}");
conf.set("fs.gravitino.client.oauth2.path", "{token_path}");
conf.set("fs.gravitino.client.oauth2.credential", "{client_id}:{client_secret}");
conf.set("fs.gravitino.client.oauth2.scope", "{scope}");
Path filesetPath = new Path("gvfs://fileset/sales/raw/events");
FileSystem fs = filesetPath.getFileSystem(conf);

Examples​

Each example assumes GVFS is configured and signed in as in the Quick start. The Java examples leave the authentication properties out, and the Python examples refer to the quick start's options dictionary as auth_options.

With the jar and core-site.xml from the quick start in place, the Hadoop shell works on any fileset path. On a Kerberos-secured Hadoop cluster, get a ticket first with kinit.

Work with a fileset from the Hadoop shell
hadoop fs -ls gvfs://fileset/sales/raw/events/
hadoop fs -mkdir gvfs://fileset/sales/raw/events/2026
hadoop fs -get gvfs://fileset/sales/raw/events/hello.txt ./hello.txt
Last updated on