Metrics and Prometheus

Gravitino records metrics for every service running in the server process: the main server and the Iceberg REST, Lance REST, and SCIM services beside it. One endpoint on the main server serves all of them in the Prometheus text format, so Prometheus scrapes a single target. The monitoring view in the UI is built from the same data.

Path	Format	Use
`/prometheus/metrics`	Prometheus text	Scraping by Prometheus
`/metrics`	Dropwizard JSON	Reading by hand

Both are on the server's port 8090 and need no credentials, so scrape them from inside the cluster rather than exposing them publicly.

Quick start

Fetch the metrics.

Fetch metrics in Prometheus form

kubectl -n "$NAMESPACE" port-forward "svc/$RELEASE-gravitino" 8090:8090 &
curl -s http://localhost:8090/prometheus/metrics | grep -v '^#' | head

Connect Prometheus with one of the two methods in Connect Prometheus.
Check the target. In the Prometheus UI, Status → Targets lists the Gravitino target as up, and a query for gravitino_server_2xx_responses_total returns a series per operation.

Connect Prometheus

With the Prometheus Operator

If your cluster runs the Prometheus Operator, for example through the kube-prometheus-stack chart, have the chart create a ServiceMonitor for the server:

my-values.yaml
gravitino:
  serviceMonitor:
    enabled: true
    path: /prometheus/metrics
    interval: 30s
    labels:
      release: prometheus

Set path to /prometheus/metrics. The chart's default, /metrics, is the JSON form, which Prometheus cannot parse. Set labels to whatever your Prometheus selects ServiceMonitors by; the release: prometheus above suits the kube-prometheus-stack chart installed as prometheus. Apply the change by running the install command from Install again.

With a scrape job

Any other Prometheus scrapes the server's Service directly. Add a job to prometheus.yml:

prometheus.yml
scrape_configs:
  - job_name: gravitino
    metrics_path: /prometheus/metrics
    scrape_interval: 30s
    static_configs:
      - targets: ['{release}-gravitino.{namespace}.svc:8090']

Metric list

The server exports several hundred series. Most come from two families that repeat one pattern many times: the metadata database's per-operation counters and the JVM's per-memory-pool gauges. The tables list every name, with patterns written once.

Counters are cumulative since the server last started, so take a rate over them for anything recent, as in rate(gravitino_server_5xx_responses_total[5m]). Durations are summaries with quantiles from 0.5 to 0.999 in a quantile label.

Requests

Each service records every operation it serves, with the operation in an operation label, such as create-table or list-schema. {service} is gravitino_server, iceberg_rest_server, or lance_rest_server. The SCIM service records no per-operation request metrics; it reports its thread pool and its health checks only, under scim_rest_server_* names.

Name	Type	What it measures
`{service}_1xx_responses_total` to `{service}_5xx_responses_total`	Counter	Responses by status class, per operation
`{service}_http_request_duration_seconds`	Summary	Request duration, per operation, by quantile
`{service}_http_request_duration_seconds_count`	Counter	Requests timed, per operation

Example request metrics
gravitino_server_2xx_responses_total{operation="create-table",} 12.0
gravitino_server_5xx_responses_total{operation="create-table",} 0.0
gravitino_server_http_request_duration_seconds{operation="create-table",quantile="0.95",} 0.041
gravitino_server_http_request_duration_seconds_count{operation="create-table",} 12.0

Health checks are recorded under names of their own, so probe traffic stays out of the request totals: gravitino_server_health_live_* and gravitino_server_health_ready_* for the main server, and iceberg_rest_server_iceberg_health_*, iceberg_rest_server_iceberg_health_live_* and iceberg_rest_server_iceberg_health_ready_* for the Iceberg REST service. Each carries the same response and duration names as the table.

Thread pools

Each service runs its own HTTP thread pool.

Name	What it measures
`{service}_http_server_busy_thread_num`	Threads handling a request now
`{service}_http_server_idle_thread_num`	Threads waiting for work
`{service}_http_server_total_thread_num`	Threads currently in the pool
`{service}_http_server_min_thread_num`	The pool's floor
`{service}_http_server_max_thread_num`	The pool's ceiling
`{service}_http_server_queued_request_num`	Requests waiting for a free thread

Busy threads at the maximum with nothing queued means the pool is fully used. Queued requests above zero means callers are waiting, and a queue that stays above zero is the saturation signal to act on.

Metadata database

The metadata database reports its connection pool, and each of its operations, such as deleteSchema or batchSetOwner, records success and failure counts and a duration.

Name	What it measures
`gravitino_relational_store_datasource_active_connections`	Connections in use
`gravitino_relational_store_datasource_idle_connections`	Connections open and available
`gravitino_relational_store_datasource_max_connections`	The pool's ceiling
`gravitino_relational_store_{operation}_success_total`	Calls that succeeded
`gravitino_relational_store_{operation}_failure_total`	Calls that failed
`gravitino_relational_store_{operation}_total`	Call duration, by quantile
`gravitino_relational_store_{operation}_total_count`	Calls timed

There are about 130 operations. Active connections held at the maximum means requests are waiting on the database, which shows up as rising latency on every operation that touches metadata.

JVM

Name	What it measures
`jvm_heap_used`, `jvm_heap_max`	Heap in use and maximum heap, in bytes
`jvm_heap_usage`	Heap in use as a fraction of the maximum
`jvm_non_heap_used`, `jvm_non_heap_max`	Memory outside the heap, in bytes
`jvm_non_heap_usage`	Non-heap in use as a fraction of the maximum
`jvm_total_used`, `jvm_total_max`	Heap and non-heap together, in bytes
`jvm_G1_Young_Generation_count`, `jvm_G1_Old_Generation_count`	Collections run
`jvm_G1_Young_Generation_time`, `jvm_G1_Old_Generation_time`	Time spent collecting, in milliseconds
`jvm_direct_used`, `jvm_direct_capacity`, `jvm_direct_count`	Direct buffers: bytes used, capacity, and buffers
`jvm_mapped_used`, `jvm_mapped_capacity`, `jvm_mapped_count`	Memory-mapped buffers: bytes used, capacity, and buffers
`jvm_pools_{pool}_used`, `jvm_pools_{pool}_max`, `jvm_pools_{pool}_usage`	Each memory pool, such as `jvm_pools_Metaspace_used`

The heap, non-heap and pool names also have _init and _committed siblings. Heap used rises and falls with every collection, so it says little on its own. The heap remaining after a collection is the number to trend, since a steady climb there is what an eventual out-of-memory failure looks like in advance. jvm_pools_Metaspace_used near its maximum is the other memory signal to watch.

Catalogs

JDBC catalogs report their connection pools and fileset catalogs their file system cache, labelled with provider, metalake, and catalog. Other catalog types report none.

Name	What it measures
`gravitino_catalog_datasource_active_connections`	A JDBC catalog's connections in use
`gravitino_catalog_datasource_idle_connections`	A JDBC catalog's connections open and available
`gravitino_catalog_datasource_max_connections`	A JDBC catalog's pool ceiling
`gravitino_catalog_filesystem_cache_hits`	File system cache hits
`gravitino_catalog_filesystem_cache_misses`	File system cache misses
`gravitino_catalog_filesystem_cache_loads_success`	Cache entries loaded
`gravitino_catalog_filesystem_cache_loads_failure`	Cache entries that failed to load
`gravitino_catalog_filesystem_cache_evictions`	Cache evictions, with one name per cause: `_COLLECTED`, `_EXPIRED`, `_EXPLICIT`, `_REPLACED`, `_SIZE`

Not exported

The server exports no CPU or uptime metric. Take those from the container or host, for example from cAdvisor through kube-prometheus-stack. Within the server, the closest signals to load are thread pool saturation and garbage collection time.

JSON names

/metrics returns the same metrics as JSON, grouped into gauges, counters, histograms, meters, and timers. JSON names use dots and dashes, as in jvm.heap.used or gravitino-server.create-table.2xx-responses; the Prometheus form replaces both with underscores and moves the operation into the operation label. With the port-forward from the Quick start running:

Fetch metrics as JSON

curl -s http://localhost:8090/metrics | jq '.gauges | keys | .[:10]'

What to alert on

Condition	Query
A thread pool is saturated	`gravitino_server_http_server_queued_request_num > 0` for 5 minutes
Operations are failing on the server side	`sum(rate(gravitino_server_5xx_responses_total[5m])) > 0`
The metadata database is the bottleneck	`gravitino_relational_store_datasource_active_connections >= gravitino_relational_store_datasource_max_connections` for 5 minutes
Memory is being retained and will run out	Heap after garbage collection climbing across hours
The server cannot reach its metadata database	The readiness check reporting DOWN

Use the same queries with iceberg_rest_server or lance_rest_server in place of gravitino_server for those services. See Health and readiness for the readiness check.

Quick start​

Connect Prometheus​

With the Prometheus Operator​

With a scrape job​

Metric list​

Requests​

Thread pools​

Metadata database​

JVM​

Catalogs​

Not exported​

JSON names​

What to alert on​

Related​