Statistics

Overview

A statistic is a named measurement attached to a table or one of its partitions. Row counts, sizes, and file counts are the obvious ones, and anything else you want to record about a dataset can sit beside them.

Statistics are how the catalog answers questions about size and shape without touching the data. That matters most on a federated estate, where asking each source system directly means a different mechanism per system.

Quick start

1. Open a table. Statistics attach to tables and to partitions of tables. See Tables and Views.

2. Read what is already there. Some statistics are maintained by Gravitino or the source system, and are visible without doing anything.

3. Record your own. A custom statistic carries whatever measurement matters to you, under a name you choose.

The statistics model

Reserved and custom

A reserved statistic is one Gravitino defines, such as row_count. The set can grow in future releases, so reserved names are not available for your own use.

A custom statistic is yours, and its name must begin with custom- to keep it clear of anything Gravitino may reserve later. Names are limited to 128 characters. Every statistic written through the API, for a table or a partition, must be a custom one.

Modifiable and not

A statistic is marked modifiable or not. One derived by the system, such as a count maintained by the source, is not something to overwrite by hand, and Gravitino rejects the attempt rather than accepting a value that the next refresh discards.

Custom statistics are modifiable, since nothing else maintains them.

Values

A value is typed rather than free text, so a row count is a number and a boolean is a boolean. A statistic can also exist with no value at all, which means it is known but not currently set.

Partition statistics

Statistics attach to partitions as well as to whole tables. Where a table is large and partitioned, per-partition measurements are what make pruning and planning decisions possible.

Partition statistics are held by Gravitino rather than by the catalog, so they work on any table, including tables in catalogs that expose no partition objects. Partition names are supplied by the caller, several partitions are written in one request, and statistics are listed over a range of partition names rather than one at a time. Where they are stored is a server setting; see Partition statistics in the server configuration.

Working with statistics in the UI

Statistics are not surfaced in the UI today. Reading and writing them goes through the API.

Permissions

Statistics follow the table they belong to. Reading them requires the privilege to read the table, and writing them requires the privilege to modify it. See Tables and Views.

Using the API

Statistics can be listed, set, and dropped over REST and through the Java client, for tables and for partitions. The Python client covers table statistics but not partition statistics. The REST reference documents each call for table statistics and partition statistics, with the client code under the Java and Python tabs.