Skip to main content

Upgrades

An upgrade is a helm upgrade to the new chart version with the same my-values.yaml you installed with. The chart handles the order. A hook Job scales the Gravitino server to zero, a schema Job brings the metadata database from its recorded schema version to the one the new image expects, and the server starts on the new image only once the recorded version matches. The UI, MCP server, metrics service, OpenSearch and Marquez restart on their new images at the same time.

Seven steps: set the variables, record what the server holds, back up the metadata database, pull the new chart, check the render, upgrade, and verify. Inside helm upgrade, a hook scales the server to zero, the schema Job migrates the metadata database, and the server starts on the new image once the versions match

Your connected sources are not touched. Gravitino stores catalog definitions, tags, policies, and grants, not your data, so an upgrade affects metadata about connections rather than anything in the systems those connections point at.

Before you start​

Read the release notes for every version between the one you are on and the one you are moving to, and the new package's chart-release-summary-{version}.md, which lists any value the new version requires. Your existing my-values.yaml carries over; add only what the summary calls for. Helm builds the upgraded release from the chart defaults plus the values you pass, so pass the same file and license key you installed with rather than relying on what is already in the cluster.

Passwords the chart generated at install, such as the built-in PostgreSQL's, are kept. You do not look them up or pass them to the upgrade.

Schedule downtime. The server is stopped from the start of the upgrade until the new version is ready, so API and UI requests fail in that window.

The schema Job runs when the metadata database is the chart's own PostgreSQL or MySQL, or an external jdbc:postgresql:// or jdbc:mysql:// database. If your DBA applies schema changes instead, with gravitino.schema.enabled: false, have the DBA apply them before step 6.

Upgrade​

Run every step in one shell. Steps 2 and 7 use a shell function and a file that step 2 creates.

  1. Set your variables. These are the variables from step 1 of Install, with VERSION and PKG pointing at the new release.

    Set the upgrade variables
    VERSION={new_chart_version}
    REGISTRY=730335553010.dkr.ecr.us-east-1.amazonaws.com
    NAMESPACE=datastrato
    RELEASE=enterprise
    PKG={path_to_new_release_package}
    ADMIN_PASSWORD='{admin_password}'
    LICENSE_FILE=$(ls "$PKG"/*license-prod-*.txt)
    KEY_FILE=$(ls "$PKG"/*ecr-pull-user_accessKeys.csv)
    ls -l "$LICENSE_FILE" "$KEY_FILE" my-values.yaml

    The ls lists your license file, the pull key and my-values.yaml. Run the commands from the directory that holds my-values.yaml.

  2. Record what the server holds. The list you save here is what step 7 compares against, so a lost or changed definition shows up as a difference rather than going unnoticed.

    Record the metadata before the upgrade
    record() {
      kubectl -n "$NAMESPACE" port-forward svc/"$RELEASE"-gravitino 8090:8090 >/dev/null 2>&1 &
      local pf=$!; sleep 3
      for m in $(curl -s -u "admin:$ADMIN_PASSWORD" http://localhost:8090/api/metalakes | jq -r '.metalakes[].name'); do
        echo "metalake $m"
        for kind in catalogs roles users groups tags policies; do
          curl -s -u "admin:$ADMIN_PASSWORD" "http://localhost:8090/api/metalakes/$m/$kind" \
            | jq -r --arg m "$m" --arg k "$kind" '(.identifiers // [])[].name, (.names // [])[] | "\($m) \($k) \(.)"'
        done
      done | sort
      kill "$pf"
    }
    record > metadata-before.txt
    wc -l metadata-before.txt

    The file has one line per metalake and one per catalog, role, user, group, tag and policy in it. The function signs in as the local admin account, which needs Basic authentication enabled. On a server that accepts only OAuth, replace each -u "admin:$ADMIN_PASSWORD" with -H "Authorization: Bearer $TOKEN" and a token for a service administrator.

  3. Back up the metadata database. The backup is what you restore from if the upgrade fails, so take it even when the upgrade looks routine. The chart's PostgreSQL holds two databases, gravitino for the metadata and marquez for lineage; dump each one, as in Backup and restore.

    Back up the metadata and lineage databases
    PG_POD="$RELEASE-gravitino-postgresql-0"
    for DB in gravitino marquez; do
      kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- \
        pg_dump -U postgres -Fc "$DB" > "$DB-$(date +%Y%m%d).dump"
    done
    ls -l ./*.dump

    Each file must be larger than zero bytes. For the chart's MySQL or a database you operate yourself, use the methods in Backup and restore. Back up the database of every Iceberg catalog with a JDBC backend at the same time, as that page describes.

  4. Pull the new chart. Read the pull key and log in to the registry as in steps 2 and 3 of Install, then pull the new version.

    Pull the new chart
    helm pull "oci://$REGISTRY/datastratocharts/enterprise-helm" --version "$VERSION"
    ls -l "enterprise-helm-$VERSION.tgz"

  5. Check the render. Rendering the new chart with your existing values reports any value the new version requires that you have not set, and any setting it refuses.

    Render the new chart against the cluster
    helm template "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
      --dry-run=server > /tmp/render.yaml && echo "RENDER OK"

    It must print RENDER OK. Otherwise fix my-values.yaml and run it again.

  6. Upgrade.

    Upgrade the release
    helm upgrade "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
      --timeout 15m 2>&1 | tee "helm-upgrade-$VERSION.log"

    Helm prints the release notes when it finishes, while the new pods are still starting. If the command is interrupted, Helm records that revision as failed; run the same command again.

    Read the schema Job's log now, since Kubernetes deletes the finished Job ten minutes later.

    Read the schema migration log
    kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/component=schema-migrate --tail=100

    The log names the recorded schema version and the target. When they differ, the Job applies the migration scripts between them. When they match, it reports that the schema is already at the target and skips the scripts, which is the normal result when the new version has no schema change.

    If the Job fails, the Gravitino pod never becomes ready and its init container gives up after ten minutes. Restore from your backup as described in Roll back and contact support with the log. A failed migration usually means the schema has drifted from what it expects, and repairing that by hand risks leaving the store in a state no later migration accepts.

  7. Verify. Confirm the rollout finished and every workload is on the new images.

    Check the rollout and the running images
    kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE"-gravitino --timeout=15m
    kubectl -n "$NAMESPACE" get pods \
      -o jsonpath='{range .items[*]}{.metadata.name}{"  "}{.spec.containers[0].image}{"\n"}{end}'

    The Gravitino, UI, MCP server, metrics, OpenSearch and Marquez images carry the new version tag. Then confirm the server is serving rather than merely running, and that every definition from step 2 is still there.

    Check readiness and compare the metadata
    kubectl -n "$NAMESPACE" exec deployment/"$RELEASE"-gravitino -c gravitino -- \
      curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health/ready
    record > metadata-after.txt
    diff metadata-before.txt metadata-after.txt && echo "METADATA UNCHANGED"

    Readiness returns 200 and the comparison prints METADATA UNCHANGED. See Health and readiness if readiness returns 503.

Roll back​

Rolling back the Helm release does not undo a schema migration. If the log in step 6 reported that the schema was already at the target, nothing in the metadata database changed, and rolling back the release is enough.

Roll back the Helm release
helm rollback "$RELEASE" -n "$NAMESPACE" --timeout 15m

If the migration applied scripts, restore the backup before rolling back, following steps 2 and 3 of Restore: scale the server, metrics and lineage workloads to zero, then drop, recreate and pg_restore the gravitino and marquez databases from the dumps taken in step 3.

Then roll the release back with the command above. The rollback restarts the stopped workloads on the previous images.

Engine connectors​

Connectors run inside your Trino, Spark, and Flink pods and are upgraded separately from the server. Check the release notes for whether the connector version you are running is supported against the server version you are moving to, and plan connector updates alongside the server upgrade rather than after it.

Last updated on