Skip to main content

Server log

The server log is where Datastrato Enterprise reports what it is doing and what went wrong: startup and configuration, catalog connections, job submissions, warnings, and errors with their stack traces. The main server and the Iceberg REST, Lance REST, and SCIM services run in one process, so they share one server log.

The server log is separate from the audit log, which records who did what to which object. Both are written to the logs volume of each server pod.

Quick start​

  1. Follow the log. It lives in a directory named for the pod, under the logs volume.

    Follow the server log
    kubectl exec -n {namespace} deploy/{release}-gravitino -c gravitino -- \
      sh -c 'tail -f /opt/gravitino/logs/$HOSTNAME/gravitino-server.log'

  2. Find warnings and errors.

    Find warnings and errors in the current file
    kubectl exec -n {namespace} deploy/{release}-gravitino -c gravitino -- \
      sh -c 'grep -E " (WARN|ERROR) " /opt/gravitino/logs/$HOSTNAME/gravitino-server.log'

  3. Or read the container output. The chart also sends the server log to the container's standard output, so kubectl logs shows it in a shorter format. Standard output does not survive a pod being replaced; the file on the logs volume does.

    Read the container output
    kubectl logs -n {namespace} deploy/{release}-gravitino -c gravitino --tail 200

Where the log lives​

Each server pod writes its own file, gravitino-server.log, in a directory named for the pod on the logs volume:

Server log paths
/opt/gravitino/logs/{pod_name}/gravitino-server.log
/opt/gravitino/logs/{sub_path}/{pod_name}/gravitino-server.log    # when logPersistence.subPath is set

Format​

Each entry starts on a new line with a timestamp to the millisecond, the level, the thread, and the location in the code that logged it, followed by the message. A stack trace continues on the lines after its entry.

File layout
%d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%l] - %msg%n
FieldContent
%dDate and time, to the millisecond, in the container's time zone
%levelTRACE, DEBUG, INFO, WARN, or ERROR
%tThread name
%lClass, method, file, and line that wrote the entry
%msgThe message

The container output uses a shorter layout, with the class name and line number in place of the full location:

Container output layout
%d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

Levels​

The server logs at INFO. Two loggers are quieter by default:

LoggerLevelWhy
org.apache.gravitino.server.web.restWARNRequest handling, which is otherwise one line per call
org.casbin.jcasbinWARNAuthorization decisions, which are noisy at INFO

Set gravitino.log4j2Properties.rootLoggerLevel to change the level of everything else, and gravitino.log4j2Properties.jcasbinLoggerLevel for the authorization engine. A change to either rolls the server pods, because the chart restarts them when the logging configuration changes.

Rotation and retention​

The file rotates every day and whenever it reaches 100 MB. Rotated files are compressed and named gravitino-server_{yyyyMMdd}.{n}.log.gz beside it. A rotated file is deleted once it is older than logMaxAge or once the rotated server files together exceed serverLogMaxTotalSize, whichever happens first, oldest first. Neither limit counts the file being written.

ValueEffectDefault
gravitino.log4j2Properties.rollingAppenderPoliciesSizeSizeSize at which the file rotates100MB
gravitino.log4j2Properties.serverLogMaxTotalSizeTotal size of rotated server archives kept on the volume2GB
gravitino.log4j2Properties.logMaxAgeAge after which rotated archives are deleted. Applies to the audit, lineage, and search logs as well30d
gravitino.log4j2Properties.rootLoggerLevelLevel of the root loggerinfo

The chart writes these values into the server's log4j2.properties. Extra appenders and loggers go in gravitino.additionalLog4j2Properties, which is where the chart adds the container output appender.

What else is on the logs volume​

The logs volume, mounted at /opt/gravitino/logs, holds more than the server log. Under each pod's directory:

FileContentRotates atArchive cap
gravitino-server.logThe server log100 MBserverLogMaxTotalSize, 2 GB
gravitino_audit.logThe audit log256 MBauditLogMaxTotalSize, 10 GB
gravitino_lineage.logLineage events, written by the log sink of lineage while lineage is on (the chart default)100 MBlineageLogMaxTotalSize, 1 GB
gravitino-search.logSearch indexing activity100 MBsearchLogMaxTotalSize, 1 GB

Outside the pod directories, jobs/staging/{metalake}/{template}/{job_id}/ holds each job's jars, output.log, and error.log. The server deletes a finished job's staging directory after gravitino.job.stagingDirKeepTimeInMs, 7 days by default.

The metrics service runs in its own pod and writes gravitino-metrics-service.log in its own pod directory, with a 1 GB archive cap. When the server's logs volume is ReadWriteOnce, the chart gives the metrics service a separate 2 Gi volume; when it is ReadWriteMany, the two share it.

Sizing the logs volume​

The logs volume is a persistent volume claim, so the logs survive a pod being replaced. Setting gravitino.logPersistence.enabled to false replaces it with a scratch volume that is lost with the pod.

The default size, 14 Gi, covers the four archive caps together (2 + 10 + 1 + 1 GB) but not the files being written or job staging. Production installs should start at 40 Gi, and raise the size whenever an archive cap is raised. When the volume is full, writes to every log on it fail, the audit log included.

ValueEffectDefault
gravitino.logPersistence.sizeSize of the logs volume14Gi
gravitino.logPersistence.accessModesAccess mode. ReadWriteMany lets the metrics service share the volumeReadWriteOnce
gravitino.logPersistence.storageClassNameStorage class. Empty uses the cluster default(empty)
gravitino.logPersistence.existingClaimAn existing claim to use instead of creating one(empty)
gravitino.logPersistence.subPathDirectory on the volume to write under, for releases that share one volume(empty)

Installation and storage requirements for the volume are covered in Install with Helm.

Last updated on