Skip to main content

Production checklist

A production deployment is the install from Install with Helm with the changes below applied to the same my-values.yaml. Each item names the setting or task and links to the page that covers it in full. Work through the list before the first production workload connects, and again before each upgrade.

Quick start​

The commands read the running release and report the settings this checklist cares about most. They use the RELEASE and NAMESPACE variables from Install with Helm.

  1. Read the release values. The output shows whether the chart deploys the entity store database, and whether authorization and the audit log are on.

    Check the entity store, authorization and audit settings
    helm get values "$RELEASE" -n "$NAMESPACE" -a -o json \
      | jq '{entityStoreInCluster: .gravitino.postgresql.enabled,
             authorization: .gravitino.authorization.enable,
             audit: .gravitino.audit.enabled}'

    A production install shows entityStoreInCluster: false, authorization: true and audit: true.

  2. Read the server's memory and probes. The memory request should equal the limit, the readiness probe should show /api/health/ready, and livenessChecksHealth should be true.

    Check server memory and probes
    kubectl -n "$NAMESPACE" get deploy "$RELEASE-gravitino" -o json \
      | jq '.spec.template.spec.containers[] | select(.name == "gravitino")
            | {resources, readiness: .readinessProbe.httpGet.path,
               livenessChecksHealth: (.livenessProbe | tostring | test("/api/health/live"))}'

  3. Check the logs volume. A production install starts at 40Gi.

    List persistent volume claims and their sizes
    kubectl -n "$NAMESPACE" get pvc

  4. Check health. The aggregate endpoint returns 200 when the server can reach its entity store.

    Check server health
    kubectl -n "$NAMESPACE" exec deploy/"$RELEASE-gravitino" -c gravitino -- \
      curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health

Checklist​

Platform​

  • Production license key. Replace the trial key with the production key from your account team, and plan renewal ahead of the expiry date. An expired license stops service with no grace period. See Licensing.
  • Images in your own registry. Mirror the chart and images with the offline toolkit and point global.registry at your registry, so node replacement and scale-out do not depend on the Datastrato release registry. See Air-gapped installation.
  • Server memory request equal to its limit. Keep requests.memory equal to limits.memory so the server is the last pod evicted under node pressure. When you raise the heap, raise MaxMetaspaceSize in GRAVITINO_MEM and the limit together, since metaspace grows with the number of catalogs and features in use. gravitino.env replaces the chart's list rather than adding to it, so keep the GRAVITINO_MEM entry when you add a variable. See Workloads and resources.
  • One server replica. Multiple replicas are not supported in this release. Availability comes from Kubernetes restarting the pod, which the health probes drive. See Before you start.
  • Logs volume of at least 40Gi. The default 14Gi covers the archive caps but not the files being written or job staging. When the volume fills, writes to every log on it fail, the audit log included. Set gravitino.logPersistence.size. See Server log.
  • Node capacity planned on limits. Pods schedule on their requests, but the server, OpenSearch and the metrics service grow toward their limits under load. The server must fit on one node. See Requirements.

Metadata and backups​

  • Entity store on a database you operate. The chart's in-cluster PostgreSQL is for evaluation. Set gravitino.postgresql.enabled: false and point the server at a managed or highly available PostgreSQL or MySQL. Decide before real metadata is loaded, since moving it later means a dump and restore. See Metadata database.
  • Lineage database decided. Marquez shares the in-cluster PostgreSQL by default. Give it an external database alongside the entity store if you use lineage. See Lineage database.
  • Scheduled backups. Back up the entity store, the lineage database, every Iceberg catalog's JDBC backend, the release values and the license file. The search index is rebuilt from the entity store and needs no backup. See Backup and restore.
  • Restore tested. Restore a backup into a separate namespace at least once, and again after any change to the database layout. A backup that has never been restored is a hypothesis. See Restore.

Security​

The items below are the minimum. Security hardening has the full list, from roles and access reviews to which hosts to publish.

  • Trusted TLS certificate on the ingress. The browser must trust the certificate for the UI hostname. With a self-signed certificate the browser discards the sign-in cookies and login fails. See Before you start.
  • TLS on every published ingress. If you publish the Iceberg REST service to engines, set ingress.iceberg.tls as well. See Configure.
  • Encryption in transit inside the cluster, if your policy requires it. The ingress terminates TLS for outside clients, and traffic from there to the server runs on the cluster network. Run the release in a service mesh with mutual TLS to encrypt those hops. See HTTPS.
  • Sign-in through your identity provider. Set gravitino.authenticators: "oauth,basic" so users sign in through your identity provider while the local admin account remains for recovery. See Authentication.
  • Admin password changed. The password in my-values.yaml seeds the admin user once at first install. Change it in the UI or with the user API, then remove it from the values file. See Local users and groups.
  • Authorization on. gravitino.authorization.enable is true by default. Leave it on, and keep gravitino.authorization.serviceAdmins to the accounts that administer the server. See Access control.
  • Users and groups provisioned from your directory. Provision through SCIM rather than creating users by hand, so leavers lose access when the directory removes them. See SCIM provisioning.
  • No credentials in the values file. Hold the license, database passwords and storage keys in Secrets you create, and name them in my-values.yaml. See Secrets.
  • Token-based credential vending. Set credential-providers explicitly on each catalog that vends storage credentials, and choose a token provider such as s3-token, gcs-token or adls-token. Token providers vend short-lived credentials scoped to the table path. Static-key providers hand every client that can load a table the catalog's long-lived key. See Credential vending.
  • Least-privilege storage credentials per catalog. Give each catalog a role or key that reaches only its own storage location, so a vended credential cannot reach another catalog's data. See Credential vending.
  • UI session limits set. The UI signs users out after 15 minutes idle and after 5 hours in total by default. Set both to match your policy. See Session timeout.

Operations​

  • Health probes on the health endpoints. The chart's readiness probe targets /api/health/ready and its liveness probe checks /api/health/live. Keep both if you override the probes, and keep liveness off any check that includes the entity store, so a database outage does not restart the server. See Health and readiness.
  • Metrics scraped. Enable gravitino.serviceMonitor with path: /prometheus/metrics; the default /metrics path is the JSON form, which Prometheus cannot parse. See Scraping on Kubernetes.
  • Alerts in place. Alert on queued requests, a rising rate of 5xx responses, entity store connections held at the maximum, heap after garbage collection climbing across hours, and readiness reporting DOWN. See What to alert on.
  • Audit log shipped off the pod. The chart keeps rotated audit files for 30 days or 10 GB, whichever comes first. Send the log to your log pipeline to keep it longer or search it across pods. See Audit log.
  • Diagnostics procedure known. Know how to collect a diagnostic bundle before you need one for a support case. See Collect diagnostics.

Upgrades​

  • Release notes read for every version in between. Read them, and the package's chart-release-summary-{version}.md, before each upgrade. See Release notes.
  • Backup taken immediately before the upgrade. The upgrade migrates the entity store schema, and a backup is the way back. See Rolling back.
  • Downtime scheduled. The server is stopped from the start of an upgrade until the new version is ready, so API and UI requests fail in that window. See Upgrades.
Last updated on