Skip to main content

Production checklist

A production deployment is the install from Install with the changes below applied to the same my-values.yaml. Each item names the setting or task and links to the page that covers it in full. Work through the list before the first production workload connects, and again before each upgrade.

Quick start​

The commands read the running release and report the settings this checklist cares about most. They use the RELEASE and NAMESPACE variables from Install.

  1. Read the release values. The output shows whether the chart deploys the metadata database, and whether authorization and the audit log are on.

    Check the metadata database, authorization and audit settings
    helm get values "$RELEASE" -n "$NAMESPACE" -a -o json \
      | jq '{entityStoreInCluster: .gravitino.postgresql.enabled,
             authorization: .gravitino.authorization.enable,
             audit: .gravitino.audit.enabled}'

    A production install shows entityStoreInCluster: false, authorization: true and audit: true.

  2. Read the server's memory and probes. The memory request should equal the limit, the readiness probe should show /api/health/ready, and livenessChecksHealth should be true.

    Check server memory and probes
    kubectl -n "$NAMESPACE" get deploy "$RELEASE-gravitino" -o json \
      | jq '.spec.template.spec.containers[] | select(.name == "gravitino")
            | {resources, readiness: .readinessProbe.httpGet.path,
               livenessChecksHealth: (.livenessProbe | tostring | test("/api/health/live"))}'

  3. Check the logs volume. A production install starts at 40Gi.

    List persistent volume claims and their sizes
    kubectl -n "$NAMESPACE" get pvc

  4. Check health. The aggregate endpoint returns 200 when the server can reach its metadata database.

    Check server health
    kubectl -n "$NAMESPACE" exec deploy/"$RELEASE-gravitino" -c gravitino -- \
      curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health

Checklist​

Platform​

  • Production license key. Replace the trial key with the production key from your account team, and plan renewal ahead of the expiry date. An expired license stops service at the end of the grace period set in the key. See Licensing.
  • Images in your own registry. Mirror the chart and images with the offline toolkit and point global.registry at your registry, so node replacement and scale-out do not depend on the Datastrato release registry. See Air-gapped installation.
  • Server memory request equal to its limit. Keep requests.memory equal to limits.memory so the server is the last pod evicted under node pressure. When you raise the heap, raise MaxMetaspaceSize in GRAVITINO_MEM and the limit together, since metaspace grows with the number of catalogs and features in use. gravitino.env replaces the chart's list rather than adding to it, so keep the GRAVITINO_MEM entry when you add a variable. See Workloads and resources.
  • Server replicas decided. One replica relies on Kubernetes restarting the pod, which the health probes drive. Several replicas keep query traffic answering while a pod restarts, with administrative changes made one at a time. See Run more than one server.
  • Logs volume of at least 40Gi. The default 14Gi covers the archive caps but not the files being written or job staging. When the volume fills, writes to every log on it fail, the audit log included. Set gravitino.logPersistence.size. See Server log.
  • Node capacity planned on limits. Pods schedule on their requests, but the server, OpenSearch and the metrics service grow toward their limits under load. The server must fit on one node. See Requirements.

Metadata and backups​

  • Metadata database you operate. The chart's in-cluster PostgreSQL is for evaluation. Set gravitino.postgresql.enabled: false and point the server at a managed PostgreSQL or MySQL with automatic failover, using the service's endpoint name. Decide before real metadata is loaded, since moving it later means a dump and restore. See High availability.
  • Lineage database decided. Marquez shares the in-cluster PostgreSQL by default. Give it an external database alongside the metadata database if you use lineage. See Lineage database.
  • Scheduled backups. Back up the metadata database, the lineage database, every Iceberg catalog's JDBC backend, the release values and the license file. The search index is rebuilt from the metadata database and needs no backup. See Backup and restore.
  • Restore tested. Restore a backup into a separate namespace at least once, and again after any change to the database layout. A backup that has never been restored is a hypothesis. See Restore.

Security​

The items below are the minimum. Security hardening has the full list, from roles and access reviews to which hosts to publish.

  • Trusted TLS certificate on the ingress. The browser must trust the certificate for the UI hostname. With a self-signed certificate the browser discards the sign-in cookies and login fails. See Before you start.
  • TLS on every published ingress. If you publish the Iceberg REST service to engines, set ingress.iceberg.tls as well. See Configure.
  • Encryption in transit inside the cluster, if your policy requires it. The ingress terminates TLS for outside clients, and traffic from there to the server runs on the cluster network. Run the release in a service mesh with mutual TLS to encrypt those hops. See HTTPS.
  • Sign-in through your identity provider. Set gravitino.authenticators: "oauth,basic" so users sign in through your identity provider while the local admin account remains for recovery. See Authentication.
  • Admin password changed. The password in my-values.yaml seeds the admin user once at first install. Change it in the UI or with the user API, then remove it from the values file. See Local users and groups.
  • Authorization on. gravitino.authorization.enable is true by default. Leave it on, and keep gravitino.authorization.serviceAdmins to the accounts that administer the server. See Access control.
  • Users and groups provisioned from your directory. Provision through SCIM rather than creating users by hand, so leavers lose access when the directory removes them. See SCIM provisioning.
  • No credentials in the values file. Hold the license, database passwords and storage keys in Secrets you create, and name them in my-values.yaml. The external metadata database password is the exception in this release: the chart takes it only from gravitino.entity.jdbcPassword. See Secrets.
  • Token-based credential vending. Set credential-providers explicitly on each catalog that vends storage credentials, and choose a token provider such as s3-token, gcs-token or adls-token. Token providers vend short-lived credentials scoped to the table path. Static-key providers hand every client that can load a table the catalog's long-lived key. See Credential vending.
  • Least-privilege storage credentials per catalog. Give each catalog a role or key that reaches only its own storage location, so a vended credential cannot reach another catalog's data. See Credential vending.
  • UI session limits set. The UI signs users out after 15 minutes idle and after 5 hours in total by default. Set both to match your policy. See Session timeout.

Operations​

  • Health probes on the health endpoints. The chart's readiness probe targets /api/health/ready and its liveness probe checks /api/health/live. Keep both if you override the probes, and keep liveness off any check that includes the metadata database, so a database outage does not restart the server. See Health and readiness.
  • Metrics scraped. Enable gravitino.serviceMonitor with path: /prometheus/metrics; the default /metrics path is the JSON form, which Prometheus cannot parse. See Connect Prometheus.
  • Alerts in place. Alert on queued requests, a rising rate of 5xx responses, metadata database connections held at the maximum, heap after garbage collection climbing across hours, and readiness reporting DOWN. See What to alert on.
  • Audit log shipped off the pod. The chart keeps rotated audit files for 30 days or 10 GB, whichever comes first. Send the log to your log pipeline to keep it longer or search it across pods. See Audit log.
  • Diagnostics procedure known. Know how to collect a diagnostic bundle before you need one for a support case. See Collect diagnostics for support.

Upgrades​

  • Release notes read for every version in between. Read them, and the package's chart-release-summary-{version}.md, before each upgrade. See Release notes.
  • Backup taken immediately before the upgrade. The upgrade migrates the metadata database schema, and a backup is the way back. See Roll back.
  • Downtime scheduled. The server is stopped from the start of an upgrade until the new version is ready, so API and UI requests fail in that window. See Upgrades.
Last updated on