Skip to main content

Metadata database

The metadata database is the relational database holding every metalake, catalog, schema, table, user, group, role, and grant the server knows. The chart deploys a PostgreSQL for it by default, on a persistent volume that survives upgrades and uninstalls, which suits a trial.

For production, use a database you operate and back up as the metadata database. Moving it out of the cluster puts the metadata on infrastructure with your own snapshots, retention, and failover, and takes the database out of the release's lifecycle.

On an external database you create the database and the user. The chart's schema Job creates the tables on install and upgrades them on every helm upgrade, as it does for the database it deploys.

The metadata database does not hold the table pointers of an Iceberg catalog. Each Iceberg catalog keeps those in a store of its own, which can be a separate database on the same server; see JDBC backend.

Quick start​

  1. Create the database and user. The chart creates neither on an external server. Make the JDBC user the owner of the database. The schema Job and the server use the user's default schema, so do not set currentSchema in the JDBC URL.

    Create the PostgreSQL database and user
    CREATE USER {db_user} WITH PASSWORD '{db_password}';
    CREATE DATABASE {database} OWNER {db_user};
    \c {database}
    GRANT ALL ON SCHEMA public TO {db_user};

    As the owner of the database, the user can create tables in public on PostgreSQL 15 and later as well; the GRANT is needed only when someone else owns the database. The user creates and alters the tables during schema upgrades and reads and writes their rows at run time, so a read-write user that does not own the tables is not enough.

  2. Make it the metadata database. Turn off the built-in database and set the connection, including the password.

    my-values.yaml
    gravitino:
      postgresql:
        enabled: false
      mysql:
        enabled: false
      entity:
        jdbcUrl: "jdbc:postgresql://{postgres_host}:5432/{database}"
        jdbcDriver: "org.postgresql.Driver"
        jdbcUser: "{db_user}"
        jdbcPassword: "{db_password}"

    The entity settings are read only when both postgresql.enabled and mysql.enabled are false. While either is true the chart derives the URL, driver, and user from the database it deploys and ignores what you put here.

    The chart writes the password to a Secret named {release}-gravitino-entity-store that it owns. Do not create that Secret yourself: Helm refuses to install over a Secret it did not create, and the chart has no setting for an existing Secret. The chart keeps the Secret's password once it exists, so a later change to entity.jdbcPassword has no effect. To rotate the password, change it in the database and in the jdbc-password key of the Secret, then restart the server pods.

    With gravitino.networkPolicy.enabled, the default egress rules reach only the in-cluster database. Add the external database host to gravitino.networkPolicy.egressRules.

  3. Apply the values. Add the blocks above to the values file you installed with, and run the install command from Install again, with the same release, namespace, and license.

    Reinstall with the new values
    helm upgrade --install "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
      --timeout 15m 2>&1 | tee helm-install.log

    The schema Job runs against the new database and the server waits for it before starting.

    The new database starts empty. Moving a release that already holds metadata to an external database means restoring a dump of the old one into it first; see Backup and restore. The in-cluster database's volume is kept, so nothing is lost if you have not.

  4. Confirm the server reached the database. See Verify the connection.

Supported databases​

DatabaseSupported versionsShipped by the chart
PostgreSQL12 to 1616.15, the default
MySQL88.4, from the Red Hat rhel10/mysql-84 image, which needs a Red Hat pull secret; the community 8.4.11 image is an option

Use MySQL​

MySQL works the same way, with a different URL and driver. MySQL Connector/J is not in the server image, so the chart must supply it at runtime. By default the chart downloads it from Maven Central when the pod starts. An air-gapped cluster sets gravitino.jdbcDrivers.download.mavenMirror to an internal Maven mirror, or sets gravitino.jdbcDrivers.mode to image or volume to supply the jar another way.

Create the MySQL database and user
CREATE USER IF NOT EXISTS '{db_user}'@'%' IDENTIFIED BY '{db_password}';
CREATE DATABASE IF NOT EXISTS {database};
GRANT ALL PRIVILEGES ON {database}.* TO '{db_user}'@'%';
FLUSH PRIVILEGES;
my-values.yaml
gravitino:
  postgresql:
    enabled: false
  mysql:
    enabled: false
  jdbcDrivers:
    enabled: true
    mysql:
      enabled: true
  entity:
    jdbcUrl: "jdbc:mysql://{mysql_host}:3306/{database}"
    jdbcDriver: "com.mysql.cj.jdbc.Driver"
    jdbcUser: "{db_user}"
    jdbcPassword: "{db_password}"

Setting gravitino.mysql.enabled: true is a different thing entirely: it deploys a MySQL inside the cluster in place of the default PostgreSQL. For an external MySQL both flags stay false.

Lineage database​

Turning off gravitino.postgresql.enabled removes the database the lineage service uses as well. Lineage keeps its own database on that same in-cluster PostgreSQL by default, so moving the metadata database out without providing for lineage leaves the lineage pods unable to start.

Two routes are open. Point lineage at an external PostgreSQL of its own, or have the chart deploy a PostgreSQL for lineage alone.

External PostgreSQL for lineage​

  1. Create the lineage database and user. Nothing provisions them on an external server.

    Create the lineage database and user
    CREATE USER buendia WITH PASSWORD '{lineage_password}';
    CREATE DATABASE marquez OWNER buendia;
    \c marquez
    GRANT ALL ON SCHEMA public TO buendia;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO buendia;

  2. Point lineage at it.

    my-values.yaml
    marquez:
      postgresql:
        enabled: false
      marquez:
        inClusterDB:
          postgresql:
            enabled: false
        db:
          host: "{postgres_host}"
          port: 5432
          name: marquez
          user: buendia
          password: "{lineage_password}"

The lineage service runs its own schema migrations at startup, so the tables are created for you once the database and user exist. The chart's default lineage password, used by both routes unless you set one, is a public default; set your own. The names above are the chart defaults and can be changed, as long as the values file and the database agree. See Lineage for what the service does with them.

Dedicated in-cluster PostgreSQL for lineage​

my-values.yaml
marquez:
  postgresql:
    enabled: true

The chart then deploys a PostgreSQL for lineage only, with its own database and user.

Schema​

The schema Job runs whenever the chart deploys the database, and whenever gravitino.entity.jdbcUrl starts with jdbc:postgresql:// or jdbc:mysql://.

EventWhat the chart does
helm installThe Job applies the base schema, then the enterprise schema, and records the version.
helm upgradeA pre-upgrade hook scales the server to zero, the Job applies the upgrade scripts, and the server returns.

The scripts come from the server image, so the schema always matches the server version the release runs.

Schema settings​

ValueDefaultPurpose
gravitino.schema.enabledunsetUnset runs the Job automatically as described above. false skips it. true requires a PostgreSQL or MySQL target.
gravitino.schema.baselineVersionemptyThe Gravitino version that created an existing database with no recorded version. Set to the image version to record it without DDL.
gravitino.schema.sslModeemptyClient TLS mode for the Job. Empty takes it from the JDBC URL.
gravitino.schema.dbHostemptyHost, when the JDBC URL is not a plain host:port/database form.
gravitino.schema.dbPortemptyPort, for the same case.
gravitino.schema.dbNameemptyDatabase name, for the same case.
gravitino.schema.wait.timeoutSeconds600How long the server waits for the Job before its init container fails.

The Job parses the JDBC URL for host, port, and database. It cannot parse IPv6 addresses, a comma-separated host list, or a Cloud SQL socket; set dbHost, dbPort, and dbName for those. The Job recognizes only URLs that start with jdbc:postgresql:// or jdbc:mysql://. A jdbc:mysql:loadbalance:// URL skips the Job entirely, so the server starts against an empty database; apply the schema yourself, as described next.

The Job does not mount certificate files, so sslMode values that verify the server certificate, verify-ca and verify-full for PostgreSQL or VERIFY_CA and VERIFY_IDENTITY for MySQL, fail. require works.

Apply the schema yourself​

Where a DBA applies DDL, turn the Job off:

my-values.yaml
gravitino:
  schema:
    enabled: false

With the Job off the chart neither creates nor upgrades the tables. Run the scripts from /opt/gravitino/scripts in the server image the release runs, against the database you created: on a fresh database the base schema, then the enterprise schema; on an upgrade, the upgrade-{old_version}-to-{new_version} script for each version step in order, with the server stopped.

EngineBase schemaEnterprise schema
PostgreSQLpostgresql/schema-{version}-postgresql.sqlenterprise/postgresql/enterprise-schema-{version}-postgresql.sql
MySQLmysql/schema-{version}-mysql.sqlenterprise/mysql/enterprise-schema-{version}-mysql.sql

Apply only the upgrade scripts to an existing database, never the base schema again. On PostgreSQL, re-running the base schema against a populated database fails.

High availability​

Kubernetes restarts a server that stops, and several replicas keep query traffic answering meanwhile. Everything the servers know is in the metadata database, so availability depends on that database. Use a database service with automatic failover to a standby, such as Amazon RDS Multi-AZ or Aurora, Azure Database for PostgreSQL or MySQL with zone-redundant high availability, or Cloud SQL with high availability.

Point jdbcUrl at the service's endpoint name, which moves to the new primary on failover, not at the address of one instance. The server reads and writes through that one address and does not use read replicas.

During a failover, requests that need the database fail with an error and the readiness check reports the database as down. The server tests each connection before using it, so once the endpoint answers again it replaces the broken connections and carries on without a restart.

The same applies to each Iceberg catalog's JDBC database, since every commit to an Iceberg table goes there, and to the lineage database if you use lineage.

Backups​

Back up the metadata database, and the lineage database if you use lineage. What else to back up, and how to restore, is covered in Backup and restore.

Verify the connection​

The Settings tab under Configure in the UI shows the Entity store group, read only, with the URL, driver, and user the server is running with. The password key shows as Write-only.

When the server cannot reach its database on a new install or upgrade, the server container never starts. The pod waits in its wait-for-schema init container until the schema Job succeeds, and fails after gravitino.schema.wait.timeoutSeconds. A missing database, the wrong user or password, and an unreachable host all show up in the schema Job's log:

Read the schema Job's log
kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/component=schema-migrate --tail 100
kubectl -n "$NAMESPACE" logs "deployment/$RELEASE-gravitino" -c wait-for-schema

With the schema Job turned off, the server itself fails at startup when it cannot reach the database, and the pod restarts; the server log names the failure.

Once the server is running, the readiness endpoint probes the metadata database. A server that lost its database returns 503 with an entityStore check in the DOWN state, and the pod goes unready:

Check readiness through a port-forward
kubectl -n "$NAMESPACE" port-forward "svc/$RELEASE-gravitino" 8090:8090 &
curl -s http://localhost:8090/api/health/ready

See Health and readiness for the response format and the probe timeout.

  • Install, for the values file and the install command these steps reuse
  • Server configuration, for the connection pool and version retention settings that apply whichever database holds the metadata
Last updated on