Skip to main content

Troubleshooting

The entries on this page are problems that appear on a running deployment after the install has succeeded. Problems during the install itself, such as render failures, image pulls and pods that never start, are covered under Troubleshoot on the install page.

Quick start​

The commands use the RELEASE and NAMESPACE variables from Install with Helm.

  1. Check health. A 200 means the server is up and can reach its entity store.

    Check server health
    kubectl -n "$NAMESPACE" exec deploy/"$RELEASE-gravitino" -c gravitino -- \
      curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health

  2. Find the failed request in the server log. A refused request is logged as Authorization failed with the user, the operation and the object, which the error returned to the caller does not always carry.

    Find authorization failures and errors
    kubectl -n "$NAMESPACE" exec deploy/"$RELEASE-gravitino" -c gravitino -- \
      sh -c 'grep -E "Authorization failed| ERROR " /opt/gravitino/logs/$HOSTNAME/gravitino-server.log | tail -20'

  3. Match the message to an entry below. If none fits, collect a diagnostic bundle for your support case with Collect diagnostics.

Symptoms​

Access and permissions​

SymptomCause and fix
403 with Current user {user} is not a member of metalake {metalake}, or the metalake does not existSigning in puts a user in the Directory but not in any metalake. Assign the user a role in the metalake, which admits them, or add them to the metalake over the REST API. The same message is returned for a metalake that does not exist, so check the name as well. See How people get into a metalake.
403 with User '{user}' is not authorized to perform operation '{operation}' on metadata '{object}'The caller lacks a privilege the operation requires. A caller who cannot see an object gets 403 whether or not the object exists, so a mistyped name produces the same error as a missing grant. A 404 comes back only to a caller entitled at the parent level, for example one who may create tables in the schema. See Evaluating a request.
An object is missing from a list or from the tree in the UILists return only the entries the caller is entitled to see and do not report the rest. Reaching an object below a catalog also needs USE_CATALOG and USE_SCHEMA on the levels above it. Grant the missing privilege, or check the object as a service administrator. See Evaluating a request.
The owner of a table gets 403 loading itOwnership of a table does not include the traversal privileges above it. Grant the owner USE_CATALOG and USE_SCHEMA. Without them the owner can still read the table's owner and transfer it over the REST API. See Ownership.

Names​

SymptomCause and fix
400 naming an object, or saying '.' is reserved as the qualified-name separatorAn object in the source system has a dot in its name. With authorization enabled, one such object makes its whole schema unlistable. Rename the object in the source system. See Known issues.
409 with Catalog name {name} already exists in metalake {metalake} when creating a catalogCatalog names are unique across all metalakes on a server, ignoring case, so a name used in one metalake cannot be reused in another. Choose a different name. See Known issues.

Engines​

SymptomCause and fix
An engine connected through Iceberg REST shows the wrong catalog, or noneThe engine named no warehouse, so the service served its default catalog. The warehouse is the Gravitino catalog name. Set the engine's warehouse to the catalog it should use. See Selecting a catalog.
Engines receive the catalog's long-lived storage key instead of a temporary credentialThe catalog has no credential-providers property, so it falls back to vending the static key, which is not scoped to the table path. Setting a role ARN alone does not enable token vending. Set credential-providers to a token provider such as s3-token. See When credential-providers is omitted.

The UI​

SymptomCause and fix
The UI signs users out while they are workingThe UI signs a user out after 15 minutes without activity and after 5 hours in total, whether or not they are active. Raise both limits to match your policy. See Session timeout.
Last updated on