Skip to main content

Relational backend storage

Overview​

The entity store is the relational database holding every metalake, catalog, schema, table, user, group, role, and grant the server knows. The chart deploys a PostgreSQL for it by default, on a persistent volume that survives upgrades and uninstalls, which suits a trial.

For production, point the entity store at a database you operate and back up. Moving it out of the cluster puts the metadata on infrastructure with your own snapshots, retention, and failover, and takes the database out of the release's lifecycle.

Two differences follow from moving it, and both are covered below. The chart no longer creates the schema, so you create it before installing, and the search index is the only other state the server keeps, which is rebuilt from the entity store rather than backed up.

Quick start​

Work through the database side first, then the values file, then reinstall.

  1. Create the database, schema, and user. The chart creates none of them on an external server. PostgreSQL addresses a schema inside the database, so create both and name both in the JDBC URL later.

    Create the database, schema, and user
    CREATE DATABASE {database};
    CREATE USER {db_user} WITH PASSWORD '{db_password}';
    \c {database}
    CREATE SCHEMA {schema} AUTHORIZATION {db_user};
    GRANT ALL ON SCHEMA {schema} TO {db_user};

  2. Load the schema. See Schema below, which has the commands. Nothing creates the tables for you once the built-in database is off, and the server fails to start against an empty database.

  3. Hold the password in a Secret. The chart reads the entity store password from a Secret named {release}-gravitino-entity-store, under the key jdbc-password. Create that Secret yourself and the chart uses it as it is, rather than generating one, which keeps the password out of my-values.yaml.

    Create the entity store Secret
    kubectl -n "$NAMESPACE" create secret generic "$RELEASE-gravitino-entity-store" \
      --from-literal=jdbc-password='{db_password}'

  4. Point the entity store at it. Turn off the built-in database and set the connection. Leave jdbcPassword unset so the Secret from step 3 is used.

    my-values.yaml
    gravitino:
      postgresql:
        enabled: false
      mysql:
        enabled: false
      entity:
        jdbcUrl: "jdbc:postgresql://{postgres_host}:5432/{database}?currentSchema={schema}"
        jdbcDriver: "org.postgresql.Driver"
        jdbcUser: "{db_user}"

    The entity settings are read only when both postgresql.enabled and mysql.enabled are false. While either is true the chart derives the URL, driver, and user from the database it deploys and ignores what you put here.

  5. Apply the values. Run the install command from Install with Helm again, with the same release, namespace, and license.

    Reinstall with the new values
    helm upgrade --install "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$PKG"/gravitino-license-prod-*.txt)" \
      --timeout 15m

  6. Confirm the server reached the database. See Verifying the Connection.

Using MySQL​

MySQL works the same way, with a different URL and driver and no schema inside the database.

Create the MySQL database and user
CREATE DATABASE {database};
CREATE USER '{db_user}'@'%' IDENTIFIED BY '{db_password}';
GRANT ALL PRIVILEGES ON {database}.* TO '{db_user}'@'%';
FLUSH PRIVILEGES;
my-values.yaml
gravitino:
  postgresql:
    enabled: false
  mysql:
    enabled: false
  entity:
    jdbcUrl: "jdbc:mysql://{mysql_host}:3306/{database}"
    jdbcDriver: "com.mysql.cj.jdbc.Driver"
    jdbcUser: "{db_user}"

Setting gravitino.mysql.enabled: true is a different thing entirely: it deploys a MySQL inside the cluster in place of the default PostgreSQL. For an external MySQL both flags stay false.

Lineage database​

Turning off gravitino.postgresql.enabled removes the database the lineage service uses as well. Lineage keeps its own database on that same in-cluster PostgreSQL by default, so moving the entity store out without providing for lineage leaves the lineage pods unable to start.

Two routes are open. Point lineage at an external PostgreSQL of its own, or leave the in-cluster PostgreSQL deployed for lineage alone and move only the entity store. The first is covered here.

  1. Create the lineage database and user. Nothing provisions them once the in-cluster database is gone.

    Create the lineage database and user
    CREATE USER buendia WITH PASSWORD '{lineage_password}';
    CREATE DATABASE marquez OWNER buendia;
    \c marquez
    GRANT ALL ON SCHEMA public TO buendia;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO buendia;

  2. Point lineage at it.

    my-values.yaml
    marquez:
      postgresql:
        enabled: false
      marquez:
        inClusterDB:
          postgresql:
            enabled: false
        db:
          host: "{postgres_host}"
          port: 5432
          name: marquez
          user: buendia
          password: "{lineage_password}"

The lineage service runs its own schema migrations at startup, so the tables are created for you once the database and user exist. The names above are the chart defaults and can be changed, as long as the values file and the database agree. See Lineage for what the service does with them.

Schema​

What the chart does with the schema depends on which database it is talking to.

Entity StoreOn InstallOn Upgrade
Deployed by the chart, the defaultA migration Job creates the schema on the empty databaseThe Job applies the upgrade scripts between the two versions
External PostgreSQL or MySQLNo Job runs. Create the schema yourselfNo Job runs. Apply the upgrade scripts yourself

The Job exists only while the chart deploys the database. Turning both database flags off removes it from the release, on first install and on every upgrade after.

Creating the schema​

Take the scripts from the server image, which is where the chart's own Job reads them from, and run them against the database you created. A fresh database takes two files in order: the base schema, then the enterprise schema.

Copy the scripts out of the server image
docker run --rm -v /tmp/gravitino-scripts:/out \
  {gravitino_image} cp -r /opt/gravitino/scripts /out

For PostgreSQL, set the search path so the tables land in your schema rather than public:

Load the PostgreSQL schema
\c {database}
SET search_path TO {schema};
\i /tmp/gravitino-scripts/scripts/postgresql/schema-{version}-postgresql.sql
\i /tmp/gravitino-scripts/scripts/enterprise/postgresql/enterprise-schema-{version}-postgresql.sql
Load the MySQL schema
mysql -h {mysql_host} -u {db_user} -p {database} \
  < /tmp/gravitino-scripts/scripts/mysql/schema-{version}-mysql.sql
mysql -h {mysql_host} -u {db_user} -p {database} \
  < /tmp/gravitino-scripts/scripts/enterprise/mysql/enterprise-schema-{version}-mysql.sql

{version} is the Gravitino version in the image, and {gravitino_image} is the server image the release runs. Both come from the release package.

Record the version the schema was created at, so a later manual upgrade knows where it started:

Record the schema version
INSERT INTO gravitino_schema_migration (id, schema_version) VALUES (1, '{version}');

Upgrading the schema​

Between two versions, apply the upgrade-{old_version}-to-{new_version} script for each version step in order, from the same scripts/postgresql/ or scripts/mysql/ directory in the new image, then update the recorded version. Stop the server before applying them. Apply the upgrade scripts only, never the snapshot file: re-running a snapshot against a populated database fails.

Backups​

Back up the entity store database. Everything the server knows lives there, and no other component holds state that cannot be regenerated.

Use your database provider's snapshots or native backup tooling, on the schedule and retention your own policy calls for. A restore puts the server back where the backup was taken.

The search index is the one other store the server writes to, and it is derived, not authoritative. After restoring the entity store into a fresh search cluster, set gravitino.search.opensearch.rebuild.enabled to true for one start, which rebuilds the index from the entity store once the server is ready. Leave it false on a cluster whose index is already current.

Verifying the connection​

The readiness endpoint probes the entity store, so it answers the question directly. A server that reached its database returns UP:

Check readiness through a port-forward
kubectl -n "$NAMESPACE" port-forward "svc/$RELEASE-gravitino" 8090:8090 &
curl -s http://localhost:8090/api/health/ready

A server that has not reached the database returns 503 with an entityStore check in the DOWN state, and the pod stays unready. The server logs name the failure:

Search the server logs for database errors
kubectl -n "$NAMESPACE" logs "deployment/$RELEASE-gravitino" | grep -i -E 'jdbc|entity'

A missing schema, the wrong user, and an unreachable host all surface here. See Health and readiness for the response format and the probe timeout.

  • Install with Helm, for the values file and the install command these steps reuse
  • Server configuration, for the connection pool and version retention settings that apply whichever database the entity store uses
Last updated on