Metrics
Gravitino records metrics for every service running in the server process: the main server and the Iceberg REST, Lance REST, and SCIM services beside it. The metrics are built on Dropwizard Metrics and exported over HTTP as JSON and as Prometheus text, and through JMX. The monitoring view in the UI is built from the same data.
Quick Start
1. Fetch the metrics. One endpoint on the main server reports every service in the process, so there is no separate endpoint to scrape per service.
curl http://{gravitino_host}:8090/prometheus/metrics
2. Point Prometheus at it.
scrape_configs:
- job_name: gravitino
metrics_path: /prometheus/metrics
static_configs:
- targets: ['{gravitino_host}:8090']
3. Read the same data as JSON when inspecting by hand. /metrics returns the Dropwizard JSON
form, grouped into gauges, counters, histograms, meters, and timers.
curl http://{gravitino_host}:8090/metrics
Reading the Numbers
Counters are cumulative since the server last started, so a count on its own says nothing about the last hour. Meters also carry one-, five-, and fifteen-minute rates, and those are what to use for anything recent. Timers report quantiles from p50 to p999.
Names in the JSON form use dots and dashes, as in jvm.heap.used. The Prometheus form replaces
both with underscores, as in jvm_heap_used, and moves the operation into a label.
The server exports no CPU or uptime metric. Take those from the container or host. Within the server, the closest signals to load are thread pool saturation and garbage collection time.
Metric Families
Requests
Every service records each operation it serves: a count of responses by status class and a
histogram of request duration. Operations are named for what they do, such as create-table or
list-schema, not for the HTTP method and path.
| JSON Name | What It Measures |
|---|---|
{service}.{operation}.{1xx,2xx,3xx,4xx,5xx}-responses | Responses by status class, with rates |
{service}.{operation}.http-request-duration-seconds | Request duration, as quantiles |
{service} is gravitino-server, iceberg-rest-server, lance-rest-server, or
scim-rest-server. In Prometheus form, the create-table operation on the main server reads:
gravitino_server_2xx_responses_total{operation="create-table",} 12.0
gravitino_server_5xx_responses_total{operation="create-table",} 0.0
gravitino_server_http_request_duration_seconds_count{operation="create-table",} 12.0
gravitino_server_http_request_duration_seconds{operation="create-table",quantile="0.95",} 0.041
Thread Pools
Each service runs its own HTTP thread pool, and its gauges show how close it is to capacity.
| JSON Name | What It Measures |
|---|---|
{service}.http-server.busy-thread.num | Threads handling a request now |
{service}.http-server.idle-thread.num | Threads waiting for work |
{service}.http-server.total-thread.num | Threads currently in the pool |
{service}.http-server.min-thread.num | The pool's floor |
{service}.http-server.max-thread.num | The pool's ceiling |
{service}.http-server.queued-request.num | Requests waiting for a free thread |
Busy threads at the maximum with nothing queued means the pool is fully used. Queued requests above zero means callers are waiting, and a queue that stays above zero is the saturation signal to act on.
Entity Store
The relational entity store reports its connection pool, and each store operation records success and failure counts and a duration timer.
| JSON Name | What It Measures |
|---|---|
gravitino-relational-store.datasource.active-connections | Connections in use |
gravitino-relational-store.datasource.idle-connections | Connections open and available |
gravitino-relational-store.datasource.max-connections | The pool's ceiling |
Active connections held at the maximum means requests are waiting on the database, which shows up as rising latency on every operation that touches metadata.
JVM
JVM metrics cover heap and non-heap memory, each memory pool including Metaspace, garbage collection
counts and time for the young and old generations, and buffer pools. They start with jvm.
Heap used rises and falls with every collection, so it says little on its own. The heap remaining after a collection is the number to trend, since a steady climb there is what an eventual out-of-memory failure looks like in advance.
Catalogs
Fileset and JDBC catalogs report metrics of their own, labelled with provider, metalake, and
catalog. Other catalog types report none.
gravitino_catalog_filesystem_cache_hits{provider="fileset",metalake="{metalake}",catalog="{catalog}",} 0.0
gravitino_catalog_filesystem_cache_misses{provider="fileset",metalake="{metalake}",catalog="{catalog}",} 0.0
gravitino_catalog_datasource_active_connections{provider="jdbc",metalake="{metalake}",catalog="{catalog}",} 0.0
gravitino_catalog_datasource_idle_connections{provider="jdbc",metalake="{metalake}",catalog="{catalog}",} 1.0
gravitino_catalog_datasource_max_connections{provider="jdbc",metalake="{metalake}",catalog="{catalog}",} 10.0
What to Alert On
| Condition | What It Usually Means |
|---|---|
| Queued requests above zero for more than a few minutes | A service's thread pool is saturated |
A rising five-minute rate of 5xx responses | Operations are failing on the server side |
| Entity store active connections held at the maximum | The metadata database is the bottleneck |
| Heap after garbage collection climbing across hours | Memory is being retained and will run out |
| The readiness check reporting DOWN | The server cannot reach its entity store |
See Health and Readiness for the readiness check.