Backup and restore

A Datastrato Enterprise install keeps its state in a few places. Some of it is authoritative and has to be backed up; the rest is derived and can be rebuilt. The commands on this page assume the default install from Install with Helm, with the shell variables set there (RELEASE, NAMESPACE, LICENSE_FILE).

What to back up

State	Where it lives by default	Back up?
Entity store: all metadata, grants, tags, policies, jobs, local users, SCIM tokens	Database `gravitino` on the in-cluster PostgreSQL	Yes
Lineage history	Database `marquez` on the same PostgreSQL	Yes, if you use lineage
Iceberg table pointers	The backend database each Iceberg catalog names in its `uri`, which you operate	Yes, with the entity store
Audit log	`gravitino_audit.log` on the logs volume	Yes, by shipping it; see Audit log
Release values and license	The Helm release record and your license file	Yes
Search index	OpenSearch	No, rebuilt from the entity store
Server, lineage and search logs; job staging	The logs volume	No
JDBC driver and Iceberg jar caches	Their own volumes	No, downloaded again

The data itself, in object storage and in the source systems your catalogs connect to, is not part of a Datastrato Enterprise backup. Protect it with those systems' own tools.

The entity store holds local user passwords and SCIM tokens as hashes, and the admin password once the admin user exists. A restore brings back the passwords and tokens as they were at backup time.

Quick start

The example backs up a default install, where the entity store and lineage databases share the in-cluster PostgreSQL.

Save the release values. The license key was passed on the command line, so keep the license file with them.
Save the values
```
helm get values "$RELEASE" -n "$NAMESPACE" -o yaml > values-backup.yaml
cp "$LICENSE_FILE" .
```

Dump the databases.

Dump the entity store and lineage databases
PG_POD="$RELEASE-gravitino-postgresql-0"
kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- \
  pg_dump -U postgres -Fc gravitino > "gravitino-$(date +%Y%m%d).dump"
kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- \
  pg_dump -U postgres -Fc marquez > "marquez-$(date +%Y%m%d).dump"
ls -l ./*.dump

A dump taken while the server runs is consistent: pg_dump reads one snapshot of the database.

Back up the Iceberg backends. For each Iceberg catalog whose backend is a JDBC database, dump that database with its own tools, as close in time to the entity store dump as you can.
Store the files off the cluster.

The entity store

In-cluster PostgreSQL

Use the commands in the Quick start. pg_dump -Fc writes one database in PostgreSQL's custom format, which restores into a new cluster as well as the old one. Avoid pg_dumpall for anything but a rollback in the same cluster: it carries the old role passwords, which no longer match the Secrets a new install generates.

In-cluster MySQL

Dump the entity store from the in-cluster MySQL
kubectl -n "$NAMESPACE" exec "$RELEASE-gravitino-mysql-0" -c mysql -- \
  sh -c 'exec mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --single-transaction --routines --databases gravitino' \
  > "gravitino-$(date +%Y%m%d).sql"

Name the database with --databases, not --all-databases, which also restores MySQL's user table and its passwords.

External database

Use your provider's snapshots or native backup tooling, on your own schedule and retention. See Metadata database for how the entity store is connected.

Pod names

The in-cluster database pod is {release}-gravitino-postgresql-0, or {release}-gravitino-mysql-0 for MySQL. When the release name itself contains gravitino, the chart drops the extra -gravitino, and the pod is {release}-postgresql-0.

Lineage

Lineage history lives in the marquez database and cannot be rebuilt from the entity store. On a default install it shares the in-cluster PostgreSQL, and the Quick start dumps it. With a dedicated or external lineage database, dump marquez from there.

Iceberg catalog backends

An Iceberg catalog's tables, and the pointer to each table's current metadata file, live in the backend the catalog names, not in the entity store. The entity store holds the catalog's connection properties and the grants, tags and owners on its tables. See Iceberg catalog.

The Iceberg REST service has no database of its own. It serves the Iceberg catalogs of its metalake, so its state is the entity store plus these backends, and backing up both covers it.

Each catalog's backend is named by its catalog-backend and uri properties, and the chart never creates it:

Backend	Where the table pointers live	How to back it up
`jdbc`	The database in `uri`, in the `iceberg_tables` and `iceberg_namespace_properties` tables	`pg_dump` or `mysqldump` of that database
`hive`	The Hive Metastore in `uri`	The metastore's own backup
`rest`	The remote Iceberg REST catalog in `uri`	That catalog's own backup

When a JDBC backend shares the in-cluster PostgreSQL, add its database to the dump loop in the Quick start:

Dump an Iceberg JDBC backend on the in-cluster PostgreSQL

kubectl -n "$NAMESPACE" exec "$RELEASE-gravitino-postgresql-0" -c postgresql -- \
  pg_dump -U postgres -Fc {backend_database} > "{backend_database}-$(date +%Y%m%d).dump"

A Paimon catalog works the same way: with the jdbc backend, dump the database in its uri alongside the entity store; with filesystem, its metadata is in the warehouse in object storage; with hive or rest, back up that metastore or catalog.

Other catalog types need nothing beyond the entity store: Hive, Glue, JDBC and Kafka catalogs keep their metadata in the source system, which is backed up with that system's own tools; fileset and model catalogs keep theirs in the entity store; and Lance and Delta tables in a generic lakehouse catalog are registered in the entity store, with their own metadata in their files in storage.

Back the entity store and the Iceberg backends up together. A restore brings each back to its own backup time, and the two only agree when those times match:

Grants, tags and owners in the entity store can name tables that were dropped or renamed after the backup. Tables created after it have none.
A backend restored to an earlier point can name metadata or data files that snapshot expiry or a purge has deleted since, and those tables can no longer be read. Pause snapshot expiry and purges while the backups run.

Audit log

The audit log is the record of who did what, and nothing rebuilds it. It lives on the logs volume, which keeps rotated files for 30 days or 10 GB by default and is deleted by helm uninstall. Ship it to your log platform as it is written, as described in Audit log; a copy taken at backup time leaves gaps.

Restore

The same procedure restores into the cluster the backup came from or into a new one.

Install the release. In a new cluster, install from the saved values and license, at the same chart version as the backup, following Install with Helm. Recreate any Secrets you created yourself before installing, such as TLS certificates or an admin Secret named in gravitino.admin.existingSecret. Wait for the server to be ready.
Stop everything that uses the databases. The server, the metrics service and the lineage service all connect to them.
Scale the database clients to zero
```
kubectl -n "$NAMESPACE" scale deploy "$RELEASE-gravitino" "$RELEASE-metrics" "$RELEASE-marquez" --replicas=0
```

Replace the databases.

Restore the entity store and lineage databases
PG_POD="$RELEASE-gravitino-postgresql-0"
BACKUP_DATE=20261017    # the date in the dump file names
for DB in gravitino marquez; do
  OWNER=$([ "$DB" = marquez ] && echo buendia || echo gravitino)
  kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- dropdb -U postgres --if-exists "$DB"
  kubectl -n "$NAMESPACE" exec "$PG_POD" -c postgresql -- createdb -U postgres -O "$OWNER" "$DB"
  kubectl -n "$NAMESPACE" exec -i "$PG_POD" -c postgresql -- \
    pg_restore -U postgres -d "$DB" < "$DB-$BACKUP_DATE.dump"
done

buendia is the chart's default lineage user; use your own if you changed it. For MySQL, pipe the .sql file into mysql -uroot -p"$MYSQL_ROOT_PASSWORD" in the MySQL pod. Restore the Iceberg backend databases to the same point now.

Bring the release back. Run the same helm upgrade --install command you installed with. It runs the schema Job, which upgrades the restored schema if the backup came from an earlier version, and sets the replicas back.

A backup from a later version than the release cannot be restored: the schema Job refuses a database newer than the server.
Rebuild the search index. The index still describes the metadata before the restore. Rebuild it with a service administrator's credentials:
Rebuild the search index
```
kubectl -n "$NAMESPACE" exec "deploy/$RELEASE-gravitino" -c gravitino -- \
  /opt/gravitino/bin/index.sh rebuild --gravitino_username=admin --gravitino_password="$ADMIN_PASSWORD"
```
Use the admin password from the backup, as the next step explains. With OAuth, pass --gravitino_token instead.
Check sign-in. The admin password is the one stored in the backup, not the one in the admin Secret, which only seeds the admin user on an empty entity store. Confirm that your identity provider's SCIM token still works; a token created after the backup is gone.

What `helm uninstall` removes

Uninstalling a release deletes:

the logs volume, with the audit log on it, and the driver and jar cache volumes;
the license Secret and the Iceberg REST and OpenSearch Secrets;
the release record, including the values and the license key passed on the command line.

It keeps the in-cluster database volumes and their Secrets, the entity store Secret, the admin Secret, and the OpenSearch volume. Back up the release values and the audit log before uninstalling.

What to back up​

Quick start​

The entity store​

In-cluster PostgreSQL​

In-cluster MySQL​

External database​

Pod names​

Lineage​

Iceberg catalog backends​

Audit log​

Restore​

What helm uninstall removes​

Related​