Skip to main content

Upgrades

An upgrade is a helm upgrade to the new chart version with the same my-values.yaml you installed with. The chart handles the order. A hook Job scales the Gravitino server to zero, a schema Job brings the metadata database from its recorded schema version to the one the new image expects, and the server starts on the new image only once the recorded version matches. The UI, MCP server, metrics service, OpenSearch and Marquez restart on their new images at the same time.

Seven steps: set the variables, record what the server holds, back up the metadata database, pull the new chart, check the render, upgrade, and verify. Inside helm upgrade, a hook scales the server to zero, the schema Job migrates the metadata database, and the server starts on the new image once the versions match

Your connected sources are not touched. Gravitino stores catalog definitions, tags, policies, and grants, not your data, so an upgrade affects metadata about connections rather than anything in the systems those connections point at.

Before you start​

Read the release notes for every version between the one you are on and the one you are moving to, and the new package's chart-release-summary-{version}.md, which lists any value the new version requires. Your existing my-values.yaml carries over; add only what the summary calls for. Helm builds the upgraded release from the chart defaults plus the values you pass, so pass the same file and license key you installed with rather than relying on what is already in the cluster.

If you no longer have my-values.yaml, recover it from the release. Helm keeps the values you installed with, and only those:

Recover my-values.yaml from the release
helm get values enterprise -n datastrato -o yaml > my-values.yaml

Do not upgrade with --reuse-values. It builds the release from the defaults of the chart you are upgrading from, not the new one, so the images stay on the old version, and a value the new chart requires is missing and the render fails.

Passwords the chart generated at install, such as the built-in PostgreSQL's, are kept. You do not look them up or pass them to the upgrade.

Schedule downtime. The server is stopped from the start of the upgrade until the new version is ready, so API and UI requests fail in that window.

The schema Job runs when the metadata database is the chart's own PostgreSQL or MySQL, or an external jdbc:postgresql:// or jdbc:mysql:// database. If your DBA applies schema changes instead, with gravitino.schema.enabled: false, have the DBA apply them before step 6.

Upgrade​

Run every step in one shell. Steps 2 and 7 use a shell function and a file that step 2 creates.

  1. Set your variables. These are the variables from step 1 of Install, with VERSION and PKG pointing at the new release and LICENSE_FILE pointing at the key you run on now. The license file in the new package is an evaluation key, and upgrading with it replaces your trial or production key.

    Set the upgrade variables
    VERSION={new_chart_version}
    REGISTRY=730335553010.dkr.ecr.us-east-1.amazonaws.com
    NAMESPACE=datastrato
    RELEASE=enterprise
    PKG={path_to_new_release_package}
    ADMIN_PASSWORD='{admin_password}'
    LICENSE_FILE={path_to_your_license_file}
    KEY_FILE=$(ls "$PKG"/*ecr-pull-user_accessKeys.csv)
    ls -l "$LICENSE_FILE" "$KEY_FILE" my-values.yaml

    The ls lists your license file, the pull key and my-values.yaml. Run the commands from the directory that holds my-values.yaml.

  2. Record what the server holds. The list you save here is what step 7 compares against, so a lost or changed definition shows up as a difference rather than going unnoticed.

    Record the metadata before the upgrade
    record() {
    kubectl -n "$NAMESPACE" port-forward svc/"$RELEASE"-gravitino 8090:8090 >/dev/null 2>&1 &
    local pf=$!; sleep 3
    for m in $(curl -s -u "admin:$ADMIN_PASSWORD" http://localhost:8090/api/metalakes | jq -r '.metalakes[].name'); do
    echo "metalake $m"
    for kind in catalogs roles users groups tags policies; do
    curl -s -u "admin:$ADMIN_PASSWORD" "http://localhost:8090/api/metalakes/$m/$kind" \
    | jq -r --arg m "$m" --arg k "$kind" '(.identifiers // [])[].name, (.names // [])[] | "\($m) \($k) \(.)"'
    done
    done | sort
    kill "$pf"
    }
    record > metadata-before.txt
    wc -l metadata-before.txt

    The file has one line per metalake and one per catalog, role, user, group, tag and policy in it. The function signs in as the local admin account, which needs Basic authentication enabled. On a server that accepts only OAuth, replace each -u "admin:$ADMIN_PASSWORD" with -H "Authorization: Bearer $TOKEN" and a token for a service administrator.

  3. Back up the metadata database. The backup is what you restore from if the upgrade fails, so take it even when the upgrade looks routine. The chart's PostgreSQL holds two databases, gravitino for the metadata and marquez for lineage; dump each one, as in Backup and restore.

    Back up the metadata and lineage databases
    PG_POD="$RELEASE-gravitino-postgresql-0"
    for DB in gravitino marquez; do
    kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- \
    pg_dump -U postgres -Fc "$DB" > "$DB-$(date +%Y%m%d).dump"
    done
    ls -l ./*.dump

    Each file must be larger than zero bytes. For the chart's MySQL or a database you operate yourself, use the methods in Backup and restore. Back up the database of every Iceberg catalog with a JDBC backend at the same time, as that page describes.

  4. Pull the new chart. Read the pull key and log in to the registry as in steps 2 and 3 of Install, then pull the new version.

    Pull the new chart
    helm pull "oci://$REGISTRY/datastratocharts/enterprise-helm" --version "$VERSION"
    ls -l "enterprise-helm-$VERSION.tgz"
  5. Check the render. Rendering the new chart with your existing values reports any value the new version requires that you have not set, and any setting it refuses.

    Render the new chart against the cluster
    helm template "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
    --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
    --dry-run=server > /tmp/render.yaml && echo "RENDER OK"

    It must print RENDER OK. Otherwise fix my-values.yaml and run it again.

  6. Upgrade.

    Upgrade the release
    helm upgrade "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
    --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
    --timeout 15m 2>&1 | tee "helm-upgrade-$VERSION.log"

    Helm prints the release notes when it finishes, while the new pods are still starting. If the command is interrupted, Helm records that revision as failed; run the same command again.

    Read the schema Job's log now, since Kubernetes deletes the finished Job ten minutes later.

    Read the schema migration log
    kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/component=schema-migrate --tail=100

    The log names the recorded schema version and the target. When they differ, the Job applies the migration scripts between them. When they match, it reports that the schema is already at the target and skips the scripts, which is the normal result when the new version has no schema change.

    If the Job fails, the Gravitino pod never becomes ready and its init container gives up after ten minutes. Restore from your backup as described in Roll back and contact support with the log. A failed migration usually means the schema has drifted from what it expects, and repairing that by hand risks leaving the store in a state no later migration accepts.

  7. Verify. Confirm the rollout finished and every workload is on the new images.

    Check the rollout and the running images
    kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE"-gravitino --timeout=15m
    kubectl -n "$NAMESPACE" get pods \
    -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[0].image}{"\n"}{end}'

    The Gravitino, UI, MCP server, metrics, OpenSearch and Marquez images carry the new version tag. Then confirm the server is serving rather than merely running, and that every definition from step 2 is still there.

    Check readiness and compare the metadata
    kubectl -n "$NAMESPACE" exec deployment/"$RELEASE"-gravitino -c gravitino -- \
    curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health/ready
    record > metadata-after.txt
    diff metadata-before.txt metadata-after.txt && echo "METADATA UNCHANGED"

    Readiness returns 200 and the comparison prints METADATA UNCHANGED. See Health and readiness if readiness returns 503.

  8. Upgrade the engine connectors. The chart does not deploy Trino, Spark, or Flink, so helm upgrade leaves the Gravitino connectors inside those engines at their old version. Upgrade them now, after the server. A connector must not be newer than the server it connects to. An older connector keeps working against the upgraded server, but without the connector fixes of the new release; see Version notes for what each release changes.

    Change the connector version where you installed the connector, as described in Connectors on Kubernetes:

    Installed fromWhat to change
    A connector image as an init container (datastrato/enterprise-trino-connector, datastrato/enterprise-spark-connector, datastrato/enterprise-flink-connector)The image tag, to the new version.
    A derived Trino, Spark, or Flink imageThe connector version in the image build. Rebuild the image and roll the engine onto it.
    Maven coordinates (the Trino zip URL, or spark.jars.packages for Spark)Every connector coordinate, to the new version. Spark and Flink take two jars, the open source runtime and the Datastrato extension, and both move together.

    The connector images on Docker Hub carry every engine and version range of a release, for example datastrato/enterprise-trino-connector:1.3.2, datastrato/enterprise-spark-connector:1.3.2, and datastrato/enterprise-flink-connector:1.3.2.

    A connector loads when the engine starts:

    • Trino loads plugins only at startup. Roll the Trino deployment so every pod starts with the new plugin. A new pod reruns its init container. A connector copied into a running container is lost when the pod restarts.
    • Spark picks up the new jars in applications submitted after the change. Running applications keep the old connector until they end.
    • Flink picks up the new jars when the job or session cluster restarts.

    Verify the connectors. For Trino, confirm the plugin version and that every catalog registered:

    Check the Trino connector
    kubectl exec -n {namespace} {trino_pod} -- ls /usr/lib/trino/plugin/gravitino | grep trino-connector
    kubectl exec -n {namespace} {trino_pod} -- trino --execute \
    "SELECT catalog_name, status, last_error FROM gravitino.system.catalog_status"

    The jar name carries the version, for example gravitino-trino-connector-473-478-1.3.2.jar. Every catalog the connector supports shows REGISTERED. A catalog that shows FAILED names the reason in last_error. The plugin path and the connector catalog name (gravitino here) follow your Trino image and configuration.

    For Spark and Flink, confirm that the jars on the application classpath carry the new version, then run a statement against a Gravitino catalog, for example SHOW NAMESPACES IN {catalog}.

Roll back​

Rolling back the Helm release does not undo a schema migration. If the log in step 6 reported that the schema was already at the target, nothing in the metadata database changed, and rolling back the release is enough.

Roll back the Helm release
helm rollback "$RELEASE" -n "$NAMESPACE" --timeout 15m

If the migration applied scripts, restore the backup before rolling back, following steps 2 and 3 of Restore: scale the server, metrics and lineage workloads to zero, then drop, recreate and pg_restore the gravitino and marquez databases from the dumps taken in step 3.

Then roll the release back with the command above. The rollback restarts the stopped workloads on the previous images.

Version notes​

Each note covers one release. If you upgrade across several releases, for example 1.3.0 to 1.3.2, apply every note between your version and the target.

1.3.1 to 1.3.2​

No change to my-values.yaml is needed. 1.3.2 adds marquez.marquez.opensearch.tls.existingSecret (default opensearch-certs) and marquez.marquez.opensearch.tls.caCert (default root-ca.pem), the CA Marquez trusts for HTTPS indexing into OpenSearch. The defaults apply when you upgrade with -f my-values.yaml. If you set opensearch.tls.existingSecret to a Secret of your own, set marquez.marquez.opensearch.tls.existingSecret to the same Secret.

Before you upgrade, check every JDBC catalog's jdbc-url. From 1.3.2 a jdbc-url must not carry a password, token or user information, such as ?password=... or user:password@host, and a catalog whose URL does fails to load. On 1.3.1, list the JDBC catalogs whose URL carries one:

Find JDBC catalogs with a password in jdbc-url
GRAVITINO=https://gravitino.example.com
METALAKE=acme
curl -s -u "$GRAVITINO_USER:$GRAVITINO_PASSWORD" "$GRAVITINO/api/metalakes/$METALAKE/catalogs?details=true" \
| python3 -c 'import json,re,sys
for c in json.load(sys.stdin)["catalogs"]:
u = (c.get("properties") or {}).get("jdbc-url", "")
if re.search(r"(password|pwd|token|secret)=|//[^/@]+:[^/@]+@", u, re.I):
print(c["name"])'

Move the login of each catalog listed into jdbc-user and jdbc-password before upgrading. The uri of a JDBC-backed Iceberg or Paimon catalog is not checked.

From 1.3.2, the Spark and Flink connectors need USE_SECRETS and INCLUDE_CREDENTIAL_SECRETS on a catalog, or ownership of the metalake, to read a JDBC catalog or a catalog that vends storage credentials, because they fetch the catalog's credentials from Gravitino. Grant both to the users who read those catalogs through Spark or Flink, and only to users trusted with the catalog's account. Trino and Iceberg REST clients are not affected. See Catalog credentials.

The chart widens the entity store's column name field to 300 characters during the upgrade, so table columns with names up to 300 characters are accepted. Other names keep their 128-character limit.

Upgrade the engine connectors (step 8) to 1.3.2 for these changes:

  • Trino: BigQuery catalogs (jdbc-bigquery) register through Trino's bigquery connector. The 1.3.1 Trino connector does not support them. Trino also needs:
    • the JVM option --add-opens=java.base/java.nio=ALL-UNNAMED in jvm.config on every Trino node. Without it, catalog_status shows "Connector 'bigquery' requires additional JVM argument(s)".
    • credentials that the Trino nodes can reach: Google Application Default Credentials of the nodes, or a service account key file mounted at the same path on every Trino node and named on the Gravitino catalog with trino.bypass.bigquery.credentials-file. The catalog's own key path (jdbc-password) is a path on the Gravitino server and is not passed to Trino.
  • Trino: an Oracle NUMBER column declared without precision reads through Trino.
  • Spark: on PostgreSQL and MySQL catalogs, CREATE TABLE ... COMMENT and ALTER TABLE ... ADD COLUMN ... DEFAULT succeed, and fixed-length binary columns (MySQL BINARY(n)) read as binary.

1.3.0 to 1.3.1​

No change to my-values.yaml is needed. The chart changes only the image tags.