Skip to main content

Health and readiness

Gravitino exposes separate liveness and readiness endpoints so that a caller can tell "restart this process" apart from "send traffic somewhere else." Liveness answers whether the server can respond at all. Readiness answers whether it can reach the metadata database, and therefore do useful work.

The endpoints follow MicroProfile Health semantics. A healthy check returns 200 and an unhealthy one returns 503, both with a JSON body naming the individual checks that ran.

Quick start​

The health paths need no credentials. They are served on the server's own port, 8090, and not through the published UI host, whose proxy passes only the metadata API. The examples forward the port from the server's Service with the RELEASE and NAMESPACE variables from Install; from inside the cluster, use http://{release}-gravitino.{namespace}.svc:8090 instead.

  1. Check liveness. Liveness returns 200 whenever an HTTP thread is able to answer and the server has not run out of memory.

    Check the liveness endpoint
    kubectl -n "$NAMESPACE" port-forward "svc/$RELEASE-gravitino" 8090:8090 &
    GRAVITINO_URL=http://localhost:8090

    curl -i "${GRAVITINO_URL}/api/health/live"

  2. Check readiness. Readiness returns 200 only when the metadata database responds.

    Check the readiness endpoint
    curl -i "${GRAVITINO_URL}/api/health/ready"

  3. Check both at once. The aggregate endpoint runs the liveness and readiness checks together and reports 503 if either fails.

    Check the aggregate health endpoint
    curl -i "${GRAVITINO_URL}/api/health"

Endpoints​

PathChecksReturns 503 when
/api/health/liveHTTP serverThe server has run out of memory
/api/health/readyMetadata databaseThe metadata database check fails
/api/healthHTTP server and metadata databaseEither check fails

Each path is also served at the root of the server, without the /api prefix, for load balancers and traffic managers that require probes at well-known locations. Such a probe has to reach the server's port directly; the chart's ingress publishes the UI, which does not serve these paths. The root aliases are /health, /health/live, /health/ready, and /health.html, and the last of these maps to the aggregate endpoint rather than to a check of its own.

The response body carries an overall status and a list of individual checks. Each check has a name, a status of up or down, and a details map that explains a failure. The check names are httpServer and entityStore.

Once the server has hit an OutOfMemoryError, all three paths return 503 with a single check named jvm and the reason OutOfMemoryError; restart required. The liveness probe then fails, so Kubernetes restarts the pod, which is the only recovery from that state.

What readiness actually tests​

The metadata database check issues an existence lookup for a metalake named gravitino_health_probe. The name is a sentinel and is not expected to exist. What matters is that the store answers rather than what it answers, so a reachable store reports UP even though the lookup finds nothing.

The lookup runs on a small dedicated thread pool rather than on the request thread, so a store that has stopped responding cannot tie up HTTP threads. The pool holds one core thread, grows to four, and queues at most twenty probes before rejecting further ones.

Configuration​

PropertyDescriptionDefault
gravitino.server.health.entityStore.probeTimeoutMsTimeout for the metadata database readiness probe, in milliseconds2000

The property has no dedicated chart value, so set it under gravitino.additionalConfigItems in my-values.yaml, as described in Set other server properties.

my-values.yaml
gravitino:
  additionalConfigItems:
    gravitino.server.health.entityStore.probeTimeoutMs: "4000"

Set the timeout above the worst-case latency of the store rather than at its typical latency. A probe that exceeds the timeout is cancelled and reported as DOWN, which takes the server out of rotation, so a value tuned too tightly turns a slow backend into an outage.

Failure reasons​

A DOWN metadata database check names the reason in its details.

ReasonMeaning
entity store not initializedThe server is still starting and the store is not available yet
timeoutThe probe exceeded the configured timeout and was cancelled
probe-rejectedThe probe queue was full, meaning probes are arriving faster than the store answers
interruptedThe probe thread was interrupted
An exception class nameThe store raised that exception

A steady stream of probe-rejected means probe traffic is outpacing the store rather than that any single probe failed, so it usually points at an aggressive probe interval or a degraded backend rather than at a configuration error.

Authentication and audit​

Health paths bypass authentication, so a probe does not need credentials and does not break when authentication is enabled on the server. They are also excluded from audit logging, so probe traffic does not fill the audit log.

Both behaviors cover the root aliases as well as the canonical paths, because a forwarded request still reports its original URI.

Kubernetes probes​

The Helm chart points both probes at the health endpoints, so an install from the chart uses the checks described on this page with no change:

  • Readiness calls /api/health/ready, so a pod whose metadata database has failed stops receiving traffic.
  • Liveness runs a command in the container that calls /api/health/live. It also treats a server that stopped because the license's node limit was exceeded as live, so that Kubernetes does not restart it in a loop.

Both start checking 20 seconds after the container starts and time out after 5 seconds. Keep the chart's probes when you override other values.

Keep the readiness timeout above the metadata database probe timeout so that Kubernetes waits for the server's own answer instead of timing out first and losing the reason for the failure.

Liveness should stay on the liveness endpoint rather than the readiness or aggregate one. Pointing liveness at a check that includes the metadata database means a database outage restarts every pod, which removes the servers that would otherwise recover when the store returns.

Last updated on