Upgrades
An upgrade is a helm upgrade to the new chart version with the same my-values.yaml you installed
with. The chart handles the order. A hook Job scales the Gravitino server to zero, a schema Job brings
the metadata database from its recorded schema version to the one the new image expects, and the server
starts on the new image only once the recorded version matches. The UI, MCP server, metrics service,
OpenSearch and Marquez restart on their new images at the same time.
Your connected sources are not touched. Gravitino stores catalog definitions, tags, policies, and grants, not your data, so an upgrade affects metadata about connections rather than anything in the systems those connections point at.
Before you start
Read the release notes for every version between the one you are on and the one you are moving to,
and the new package's chart-release-summary-{version}.md, which lists any value the new version
requires. Your existing my-values.yaml carries over; add only what the summary calls for. Helm
builds the upgraded release from the chart defaults plus the values you pass, so pass the same file
and license key you installed with rather than relying on what is already in the cluster.
If you no longer have my-values.yaml, recover it from the release. Helm keeps the values you
installed with, and only those:
helm get values enterprise -n datastrato -o yaml > my-values.yaml
Do not upgrade with --reuse-values. It builds the release from the defaults of the chart you are
upgrading from, not the new one, so the images stay on the old version, and a value the new chart
requires is missing and the render fails.
Passwords the chart generated at install, such as the built-in PostgreSQL's, are kept. You do not look them up or pass them to the upgrade.
Schedule downtime. The server is stopped from the start of the upgrade until the new version is ready, so API and UI requests fail in that window.
The schema Job runs when the metadata database is the chart's own PostgreSQL or MySQL, or an external
jdbc:postgresql:// or jdbc:mysql:// database. If your DBA applies schema changes instead, with
gravitino.schema.enabled: false, have the DBA apply them before step 6.
Upgrade
Run every step in one shell. Steps 2 and 7 use a shell function and a file that step 2 creates.
-
Set your variables. These are the variables from step 1 of Install, with
VERSIONandPKGpointing at the new release andLICENSE_FILEpointing at the key you run on now. The license file in the new package is an evaluation key, and upgrading with it replaces your trial or production key.Set the upgrade variablesVERSION={new_chart_version}
REGISTRY=730335553010.dkr.ecr.us-east-1.amazonaws.com
NAMESPACE=datastrato
RELEASE=enterprise
PKG={path_to_new_release_package}
ADMIN_PASSWORD='{admin_password}'
LICENSE_FILE={path_to_your_license_file}
KEY_FILE=$(ls "$PKG"/*ecr-pull-user_accessKeys.csv)
ls -l "$LICENSE_FILE" "$KEY_FILE" my-values.yamlThe
lslists your license file, the pull key andmy-values.yaml. Run the commands from the directory that holdsmy-values.yaml. -
Record what the server holds. The list you save here is what step 7 compares against, so a lost or changed definition shows up as a difference rather than going unnoticed.
Record the metadata before the upgraderecord() {
kubectl -n "$NAMESPACE" port-forward svc/"$RELEASE"-gravitino 8090:8090 >/dev/null 2>&1 &
local pf=$!; sleep 3
for m in $(curl -s -u "admin:$ADMIN_PASSWORD" http://localhost:8090/api/metalakes | jq -r '.metalakes[].name'); do
echo "metalake $m"
for kind in catalogs roles users groups tags policies; do
curl -s -u "admin:$ADMIN_PASSWORD" "http://localhost:8090/api/metalakes/$m/$kind" \
| jq -r --arg m "$m" --arg k "$kind" '(.identifiers // [])[].name, (.names // [])[] | "\($m) \($k) \(.)"'
done
done | sort
kill "$pf"
}
record > metadata-before.txt
wc -l metadata-before.txtThe file has one line per metalake and one per catalog, role, user, group, tag and policy in it. The function signs in as the local
adminaccount, which needs Basic authentication enabled. On a server that accepts only OAuth, replace each-u "admin:$ADMIN_PASSWORD"with-H "Authorization: Bearer $TOKEN"and a token for a service administrator. -
Back up the metadata database. The backup is what you restore from if the upgrade fails, so take it even when the upgrade looks routine. The chart's PostgreSQL holds two databases,
gravitinofor the metadata andmarquezfor lineage; dump each one, as in Backup and restore.Back up the metadata and lineage databasesPG_POD="$RELEASE-gravitino-postgresql-0"
for DB in gravitino marquez; do
kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- \
pg_dump -U postgres -Fc "$DB" > "$DB-$(date +%Y%m%d).dump"
done
ls -l ./*.dumpEach file must be larger than zero bytes. For the chart's MySQL or a database you operate yourself, use the methods in Backup and restore. Back up the database of every Iceberg catalog with a JDBC backend at the same time, as that page describes.
-
Pull the new chart. Read the pull key and log in to the registry as in steps 2 and 3 of Install, then pull the new version.
Pull the new charthelm pull "oci://$REGISTRY/datastratocharts/enterprise-helm" --version "$VERSION"
ls -l "enterprise-helm-$VERSION.tgz" -
Check the render. Rendering the new chart with your existing values reports any value the new version requires that you have not set, and any setting it refuses.
Render the new chart against the clusterhelm template "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
--set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
--dry-run=server > /tmp/render.yaml && echo "RENDER OK"It must print
RENDER OK. Otherwise fixmy-values.yamland run it again. -
Upgrade.
Upgrade the releasehelm upgrade "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
--set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
--timeout 15m 2>&1 | tee "helm-upgrade-$VERSION.log"Helm prints the release notes when it finishes, while the new pods are still starting. If the command is interrupted, Helm records that revision as failed; run the same command again.
Read the schema Job's log now, since Kubernetes deletes the finished Job ten minutes later.
Read the schema migration logkubectl -n "$NAMESPACE" logs -l app.kubernetes.io/component=schema-migrate --tail=100The log names the recorded schema version and the target. When they differ, the Job applies the migration scripts between them. When they match, it reports that the schema is already at the target and skips the scripts, which is the normal result when the new version has no schema change.
If the Job fails, the Gravitino pod never becomes ready and its init container gives up after ten minutes. Restore from your backup as described in Roll back and contact support with the log. A failed migration usually means the schema has drifted from what it expects, and repairing that by hand risks leaving the store in a state no later migration accepts.
-
Verify. Confirm the rollout finished and every workload is on the new images.
Check the rollout and the running imageskubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE"-gravitino --timeout=15m
kubectl -n "$NAMESPACE" get pods \
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[0].image}{"\n"}{end}'The Gravitino, UI, MCP server, metrics, OpenSearch and Marquez images carry the new version tag. Then confirm the server is serving rather than merely running, and that every definition from step 2 is still there.
Check readiness and compare the metadatakubectl -n "$NAMESPACE" exec deployment/"$RELEASE"-gravitino -c gravitino -- \
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8090/api/health/ready
record > metadata-after.txt
diff metadata-before.txt metadata-after.txt && echo "METADATA UNCHANGED"Readiness returns
200and the comparison printsMETADATA UNCHANGED. See Health and readiness if readiness returns503. -
Upgrade the engine connectors. The chart does not deploy Trino, Spark, or Flink, so
helm upgradeleaves the Gravitino connectors inside those engines at their old version. Upgrade them now, after the server. A connector must not be newer than the server it connects to. An older connector keeps working against the upgraded server, but without the connector fixes of the new release; see Version notes for what each release changes.Change the connector version where you installed the connector, as described in Connectors on Kubernetes:
Installed from What to change A connector image as an init container ( datastrato/enterprise-trino-connector,datastrato/enterprise-spark-connector,datastrato/enterprise-flink-connector)The image tag, to the new version. A derived Trino, Spark, or Flink image The connector version in the image build. Rebuild the image and roll the engine onto it. Maven coordinates (the Trino zip URL, or spark.jars.packagesfor Spark)Every connector coordinate, to the new version. Spark and Flink take two jars, the open source runtime and the Datastrato extension, and both move together. The connector images on Docker Hub carry every engine and version range of a release, for example
datastrato/enterprise-trino-connector:1.3.2,datastrato/enterprise-spark-connector:1.3.2, anddatastrato/enterprise-flink-connector:1.3.2.A connector loads when the engine starts:
- Trino loads plugins only at startup. Roll the Trino deployment so every pod starts with the new plugin. A new pod reruns its init container. A connector copied into a running container is lost when the pod restarts.
- Spark picks up the new jars in applications submitted after the change. Running applications keep the old connector until they end.
- Flink picks up the new jars when the job or session cluster restarts.
Verify the connectors. For Trino, confirm the plugin version and that every catalog registered:
Check the Trino connectorkubectl exec -n {namespace} {trino_pod} -- ls /usr/lib/trino/plugin/gravitino | grep trino-connector
kubectl exec -n {namespace} {trino_pod} -- trino --execute \
"SELECT catalog_name, status, last_error FROM gravitino.system.catalog_status"The jar name carries the version, for example
gravitino-trino-connector-473-478-1.3.2.jar. Every catalog the connector supports showsREGISTERED. A catalog that showsFAILEDnames the reason inlast_error. The plugin path and the connector catalog name (gravitinohere) follow your Trino image and configuration.For Spark and Flink, confirm that the jars on the application classpath carry the new version, then run a statement against a Gravitino catalog, for example
SHOW NAMESPACES IN {catalog}.
Roll back
Rolling back the Helm release does not undo a schema migration. If the log in step 6 reported that the schema was already at the target, nothing in the metadata database changed, and rolling back the release is enough.
helm rollback "$RELEASE" -n "$NAMESPACE" --timeout 15m
If the migration applied scripts, restore the backup before rolling back, following steps 2 and 3 of
Restore: scale the server, metrics and lineage workloads to zero,
then drop, recreate and pg_restore the gravitino and marquez databases from the dumps taken in
step 3.
Then roll the release back with the command above. The rollback restarts the stopped workloads on the previous images.
Version notes
Each note covers one release. If you upgrade across several releases, for example 1.3.0 to 1.3.2, apply every note between your version and the target.
1.3.1 to 1.3.2
No change to my-values.yaml is needed. 1.3.2 adds marquez.marquez.opensearch.tls.existingSecret
(default opensearch-certs) and marquez.marquez.opensearch.tls.caCert (default root-ca.pem), the
CA Marquez trusts for HTTPS indexing into OpenSearch. The defaults apply when you upgrade with
-f my-values.yaml. If you set opensearch.tls.existingSecret to a Secret of your own, set
marquez.marquez.opensearch.tls.existingSecret to the same Secret.
Before you upgrade, check every JDBC catalog's jdbc-url. From 1.3.2 a jdbc-url must not carry a
password, token or user information, such as ?password=... or user:password@host, and a catalog
whose URL does fails to load. On 1.3.1, list the JDBC catalogs whose URL carries one:
GRAVITINO=https://gravitino.example.com
METALAKE=acme
curl -s -u "$GRAVITINO_USER:$GRAVITINO_PASSWORD" "$GRAVITINO/api/metalakes/$METALAKE/catalogs?details=true" \
| python3 -c 'import json,re,sys
for c in json.load(sys.stdin)["catalogs"]:
u = (c.get("properties") or {}).get("jdbc-url", "")
if re.search(r"(password|pwd|token|secret)=|//[^/@]+:[^/@]+@", u, re.I):
print(c["name"])'
Move the login of each catalog listed into jdbc-user and jdbc-password before upgrading. The uri
of a JDBC-backed Iceberg or Paimon catalog is not checked.
From 1.3.2, the Spark and Flink connectors need USE_SECRETS and INCLUDE_CREDENTIAL_SECRETS on a
catalog, or ownership of the metalake, to read a JDBC catalog or a catalog that vends storage
credentials, because they fetch the catalog's credentials from Gravitino. Grant both to the users who
read those catalogs through Spark or Flink, and only to users trusted with the catalog's account.
Trino and Iceberg REST clients are not affected. See
Catalog credentials.
The chart widens the entity store's column name field to 300 characters during the upgrade, so table columns with names up to 300 characters are accepted. Other names keep their 128-character limit.
Upgrade the engine connectors (step 8) to 1.3.2 for these changes:
- Trino: BigQuery catalogs (
jdbc-bigquery) register through Trino'sbigqueryconnector. The 1.3.1 Trino connector does not support them. Trino also needs:- the JVM option
--add-opens=java.base/java.nio=ALL-UNNAMEDinjvm.configon every Trino node. Without it,catalog_statusshows "Connector 'bigquery' requires additional JVM argument(s)". - credentials that the Trino nodes can reach: Google Application Default Credentials of the nodes,
or a service account key file mounted at the same path on every Trino node and named on the
Gravitino catalog with
trino.bypass.bigquery.credentials-file. The catalog's own key path (jdbc-password) is a path on the Gravitino server and is not passed to Trino.
- the JVM option
- Trino: an Oracle
NUMBERcolumn declared without precision reads through Trino. - Spark: on PostgreSQL and MySQL catalogs,
CREATE TABLE ... COMMENTandALTER TABLE ... ADD COLUMN ... DEFAULTsucceed, and fixed-length binary columns (MySQLBINARY(n)) read as binary.
1.3.0 to 1.3.1
No change to my-values.yaml is needed. The chart changes only the image tags.