Skip to main content

Custom partition storage

Partition statistics use a pluggable storage backend, set with gravitino.stats.partition.storageFactoryClass. Two backends ship with the server:

FactoryStorage
org.apache.gravitino.stats.storage.JdbcPartitionStatisticStorageFactoryThe default. Reuses the metadata database.
org.apache.gravitino.stats.storage.LancePartitionStatisticStorageFactoryLance datasets.

A backend of your own implements the storage factory interface and runs inside the server, so it needs a custom server image; see Deploy a custom factory.

Options​

The server passes the factory an options map. It starts from the metadata database connection, as jdbcUrl, jdbcDriver, jdbcUser, jdbcPassword, poolMaxSize, and connectionTimeoutMs, then adds every property named gravitino.stats.partition.storageOption.{key} under {key}, overriding a metadata database value of the same name.

my-values.yaml
gravitino:
  additionalConfigItems:
    gravitino.stats.partition.storageFactoryClass: "{factory_class}"
    gravitino.stats.partition.storageOption.{key}: "{value}"

Implement a custom partition storage​

Implement org.apache.gravitino.stats.storage.PartitionStatisticStorageFactory, whose create method returns your PartitionStatisticStorage, and set gravitino.stats.partition.storageFactoryClass to the factory's class name.

MyPartitionStatsStorageFactory.java
public class MyPartitionStatsStorageFactory implements PartitionStatisticStorageFactory {
    @Override
    public PartitionStatisticStorage create(Map<String, String> options) {
        // Create your custom PartitionStatsStorage here
        return new MyPartitionStatsStorage(...);
    }
}
MyPartitionStatsStorage.java
public class MyPartitionStatsStorage implements PartitionStatisticStorage {

    @Override
    public void close() throws IOException {
        // Close your storage here
    }

    @Override
    public void updateStatistics(String metalake, List<MetadataObjectStatisticsUpdate> updates)
            throws IOException {
        // Update partition statistics in your storage here
    }

    @Override
    public List<PersistedPartitionStatistics> listStatistics(
            String metalake, MetadataObject metadataObject, PartitionRange range)
            throws IOException {
        // List partition statistics from your storage here
        return Lists.newArrayList();
    }

    @Override
    public int dropStatistics(String metalake, List<MetadataObjectStatisticsDrop> drops)
            throws IOException {
        // Drop partition statistics from your storage here, returning the number actually dropped
        return drops.size();
    }
}

Deploy a custom factory​

The server loads the factory class by name from its own classpath at startup, and fails to start if the class is missing. The chart has no value for adding a jar to the server, so a custom factory needs a custom server image with the jar on its classpath.

Last updated on