Skip to main content

Install

Datastrato Enterprise installs from a Helm chart. The chart and every image it deploys are served from the Datastrato release registry on Amazon ECR, and one pull-only credential covers both. Follow the steps in order. Each one ends with something you can check before moving on, so a problem shows up at the step that caused it rather than three steps later.

The install steps are the same on every Kubernetes cluster. What differs is how the cluster meets the requirements under Before you start. Prepare the cluster covers the five tested platforms: k3s on your own hosts, Amazon EKS, Azure AKS, Google GKE, and Red Hat OpenShift.

Expect thirty minutes, most of it waiting for images to pull on the first install.

Before you start​

You need these tools on the machine you install from: kubectl pointed at the target cluster, helm 3, curl, jq, openssl, and the AWS CLI (aws). The AWS CLI is needed on every platform, not only on AWS, because the Datastrato release registry is on Amazon ECR and step 3 uses it to turn the pull key into a registry login. An air-gapped install that pulls from your own registry does not need it.

You need these files from Datastrato. Your account team sends a release package as a zip file. Unzip it into a directory of its own; step 1 calls that directory PKG.

Unpack the release package
unzip datastrato-enterprise-{version}-artifacts.zip -d datastrato-enterprise-{version}
ls datastrato-enterprise-{version}

The package contains:

FileWhat it is
datastrato-license-prod-{date}.txtYour license key
datastrato-ecr-pull-user_accessKeys.csvA pull-only AWS access key for the release registry; keep private
offline-toolkit-{version}.tar.gzImage mirroring tool, needed only for an air-gapped install
chart-release-summary-{version}.mdWhat changed in this release

Your cluster needs these. See Requirements for the supported Kubernetes and Helm versions.

  • vm.max_map_count set to at least 262144 on every node. OpenSearch will not start without it.
  • An ingress controller, and its class name (for example traefik or nginx). Find it with kubectl get ingressclass.
  • A DNS hostname that resolves to that ingress controller, for the UI. Each further service you publish, such as the REST API or Iceberg REST in Configure, needs a hostname of its own, resolving to the same controller.
  • A TLS certificate for that hostname, either as a Kubernetes TLS Secret or issued by cert-manager. Use a certificate the browser trusts. With a self-signed certificate the browser discards the sign-in cookies and login fails in a way that looks like a credential problem.
  • A default StorageClass, since the built-in PostgreSQL and OpenSearch each claim a persistent volume. Check with kubectl get storageclass.
  • Room for the workloads in Workloads and resources: about 2 cores and 14Gi of memory requested, and a node with 8Gi of allocatable memory free for the Gravitino server. One server replica is the default; see Run more than one server.

Prepare the cluster shows how to provide each of these on k3s, Amazon EKS, Azure AKS, Google GKE and Red Hat OpenShift.

Workloads and resources​

An install runs eight workloads, each a single pod with one container. The table gives what each pod requests, which a node must have free before the pod is scheduled, and its limit, the most it can use. The Gravitino server row is at the 8Gi that step 5 of the install sets; the chart's own default is 4Gi.

The chart's workloads in one namespace: the Gravitino server with its REST API, Iceberg REST and Lance REST services, PostgreSQL, OpenSearch, the UI, the MCP server, the metrics service, and Marquez, with the Ingress in front and images pulled from the Datastrato release registry

WorkloadPodCPU request / limitMemory request / limitPersistent volumes
Gravitino server{release}-gravitino-*500m / 28Gi / 8GiLogs 14Gi, JDBC drivers 2Gi, Iceberg jars 1Gi
PostgreSQL, the metadata database, also used by Marquez{release}-gravitino-postgresql-0250m / none256Mi / 1GiData 8Gi
OpenSearch, the search indexopensearch-cluster-master-0500m / 24Gi / 6GiData 8Gi
UI{release}-ui-*100m / 200m128Mi / 192MiNone
MCP server{release}-mcp-*100m / 500m256Mi / 512MiNone
Metrics service{release}-metrics-*250m / 1512Mi / 4GiLogs 2Gi
Marquez, the lineage service{release}-marquez-*200m / 500m500Mi / 750MiNone
Marquez web{release}-marquez-web-*100m / 200m128Mi / 192MiNone
Total2 / 6.4, plus PostgreSQL unboundedabout 14Gi / 21Gi35Gi

Every pod schedules once the nodes have the requests free between them. Plan capacity on the limits, since the server, OpenSearch and the metrics service grow toward theirs under load. The Gravitino server must fit on one node, so a node with 8 GiB of memory in total is too small for it: part of every node's memory is reserved for the system, leaving less than 8Gi allocatable.

Three short-lived Jobs also run. The schema Job runs at each install and upgrade, a hook Job scales the server to zero before each upgrade, and the registry-token Job renews the image pull token every six hours, requesting 100m and 128Mi with a 256Mi limit. Each finishes in seconds.

Prepare the cluster​

Follow the subsection for your platform, then continue with Install. Every platform except OpenShift issues certificates the same way, described in Issue certificates with cert-manager.

k3s on your own hosts​

k3s supplies two of the requirements itself: Traefik as the ingress controller, with class name traefik, listening on ports 80 and 443 of each node, and the local-path StorageClass as the default.

Install k3s on the first node and give your user a copy of its kubeconfig. k3s writes the original as readable by root only.

Install k3s and copy the kubeconfig
curl -sfL https://get.k3s.io | sh -
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$USER" ~/.kube/config
kubectl get ingressclass
kubectl get storageclass

The first list shows traefik and the second shows local-path (default).

Set vm.max_map_count on every node, and persist it so it survives a reboot.

Set vm.max_map_count persistently
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-opensearch.conf

Point DNS at the node. Create an A record from your UI hostname to the public IP address of the node, or of a load balancer in front of the nodes. Allow inbound ports 80 and 443 in the host firewall or security group. Port 80 must stay open if cert-manager issues the certificate, since Let's Encrypt validates the hostname over it.

local-path volumes live on one node's disk, so the PostgreSQL and OpenSearch pods can only run on the node that holds their data. On a multi-node k3s cluster, that node is a single point of failure for the metadata database unless you use your own database; see Prepare for production.

Amazon EKS​

An EKS cluster provides the control plane and the nodes, and none of the other requirements. It has no ingress controller, no working default StorageClass until the EBS CSI driver is added, and its managed nodes take no sysctl changes by hand. The steps below supply each one. Run them from the machine you install from, with eksctl installed alongside the tools under Before you start.

Set the cluster variables
CLUSTER={cluster_name}
REGION={aws_region}
AZ_NODES={availability_zone_for_nodes}
AZ_SECOND={second_availability_zone}

  1. Check the AWS account. The AWS CLI and eksctl act as the AWS identity your shell is signed in with. That identity needs permission to create an EKS cluster and node group, IAM roles, and an IAM OIDC provider, which eksctl uses to give the EBS CSI driver its IAM role. An account whose organization denies IAM OIDC provider actions cannot create the cluster below. The account must also be able to launch the node instance type.

    Check the identity, instance type and vCPU quota
    aws sts get-caller-identity
    aws ec2 describe-instance-type-offerings --region "$REGION" --location-type availability-zone \
      --filters Name=instance-type,Values=m6i.2xlarge Name=location,Values="$AZ_NODES" \
      --query 'InstanceTypeOfferings[].Location' --output text
    aws service-quotas get-service-quota --region "$REGION" --service-code ec2 --quota-code L-1216C47A \
      --query 'Quota.Value' --output text

    The first command names the account and identity, the second prints your node zone, and the quota for on-demand standard instances is at least 24, the vCPUs of three m6i.2xlarge nodes. An account on the AWS Free Plan cannot launch m6i.2xlarge at all. There, and with too little quota, the node group stays in CREATING with no error until eksctl gives up after about 25 minutes.

  2. Create the cluster. For a new evaluation cluster:

    cluster.yaml
    cat > cluster.yaml << EOF
    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    metadata:
      name: $CLUSTER
      region: $REGION
      version: "1.34"
    availabilityZones: [$AZ_NODES, $AZ_SECOND]
    autoModeConfig:
      enabled: false
    iam:
      withOIDC: true
    addons:
      - name: vpc-cni
      - name: coredns
      - name: kube-proxy
      - name: eks-pod-identity-agent
      - name: aws-ebs-csi-driver
        wellKnownPolicies:
          ebsCSIController: true
    managedNodeGroups:
      - name: ng-1
        instanceType: m6i.2xlarge
        availabilityZones: [$AZ_NODES]
        desiredCapacity: 3
        minSize: 3
        maxSize: 4
        volumeSize: 100
    EOF
    eksctl create cluster -f cluster.yaml 2>&1 | tee create-cluster.log
    kubectl get nodes

    Creation takes about twenty minutes. eksctl switches your kubectl context to the new cluster, and kubectl get nodes shows three nodes Ready. The file was tested with eksctl 0.230.0. autoModeConfig keeps the managed node group and add-ons that a later eksctl release stops creating by default, and eksctl also installs the metrics-server add-on, which the file does not list.

    If you already have a cluster, skip the file and confirm the cluster has the EBS CSI driver add-on instead:

    Check the EBS CSI driver add-on
    aws eks describe-addon --cluster-name "$CLUSTER" --region "$REGION" --addon-name aws-ebs-csi-driver \
      --query 'addon.status' --output text

    It must print ACTIVE. ResourceNotFoundException means the cluster name or region is wrong, or the add-on is missing; install the add-on with its IAM role as described in the Amazon EKS documentation for the Amazon EBS CSI driver.

    The node group sits in a single Availability Zone on purpose. EBS volumes are zonal: the PostgreSQL and OpenSearch volumes are created in one zone, and their pods can run only on nodes in that zone. A node group spread across zones can reschedule those pods where their volumes cannot attach. If you run nodes in several zones, keep a node group in every zone that holds a volume. Three m6i.2xlarge nodes run the evaluation install with room to spare; see Workloads and resources for what it needs.

  3. Make gp3 the default StorageClass.

    gp3.yaml
    cat > gp3.yaml << 'EOF'
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: gp3
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: ebs.csi.aws.com
    parameters:
      type: gp3
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
    EOF
    kubectl apply -f gp3.yaml
    kubectl annotate storageclass gp2 storageclass.kubernetes.io/is-default-class=false --overwrite 2>/dev/null
    kubectl get storageclass

    Only gp3 shows (default).

  4. Install an ingress controller. The example runs Traefik behind a Network Load Balancer. An NLB keeps a fixed address in each zone for its lifetime. Cross-zone load balancing lets the address in every zone reach the nodes, which here sit in one zone.

    traefik-values.yaml
    cat > traefik-values.yaml << 'EOF'
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    ingressClass:
      enabled: true
      isDefaultClass: true
    EOF
    helm repo add traefik https://traefik.github.io/charts
    helm repo update
    helm install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml
    kubectl -n traefik get pods

    The Traefik pod shows 1/1 Running. Use traefik as INGRESS_CLASS in step 1 of the install.

  5. Point DNS at the load balancer.

    Read the load balancer hostname
    until LB_HOST=$(kubectl -n traefik get svc traefik -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') \
      && [ -n "$LB_HOST" ]; do sleep 10; done
    echo "$LB_HOST"

    Create a CNAME record, or a Route 53 alias record, from your UI hostname to that name. A new NLB name takes a few minutes to resolve. Without a domain of your own, use an sslip.io name instead, as described under Choosing UI_HOST in step 1 of the install.

    Check the path end to end
    curl -s -o /dev/null -w '%{http_code}\n' http://{ui_hostname}/

    It prints 404. The answer comes from Traefik, which is reachable and has no routes yet.

  6. Set vm.max_map_count on every node. Apply the DaemonSet in Set vm.max_map_count with a DaemonSet.

  7. Issue certificates. Follow Issue certificates with cert-manager.

Image pulls on EKS. The node IAM role cannot pull from the Datastrato release registry, so the pull Secret from install step 4 is required on EKS exactly as on any other cluster. The chart pulls from the release registry in us-east-1 by default, and global.ecrRegion switches it to the replica in eu-west-1 or ap-northeast-1. The server image is about 3 GB, and every node without the image repeats the pull, which on EKS includes node replacement, managed node group upgrades and scale-out. For production on EKS, mirror the images into a registry in your own region; see Air-gapped installation.

Azure AKS​

AKS needs less preparation than EKS. Its Azure Disk driver is installed with a default StorageClass already set, and a load balancer Service receives a plain public IP address, so no load balancer annotations are needed. What remains is the ingress controller and vm.max_map_count. Run the steps from the machine you install from, with the Azure CLI (az) installed alongside the tools under Before you start.

Set the cluster variables
RG={resource_group}
AKS={cluster_name}
LOC={azure_region}

For a new cluster, choose a name for a new resource group; step 2 creates it. To use an existing cluster, set the group it is in.

  1. Check the VM size and vCPU quota. The cluster in step 2 uses two Standard_E4bds_v5 nodes, 4 vCPUs and 32 GB each. Any size with 4 vCPUs and 32 GB works in its place; set SIZE and FAMILY to match the size you choose. The first provider registration and az vm list-skus can each take a few minutes.

    Check the VM size and vCPU quota
    SIZE=Standard_E4bds_v5
    FAMILY=EBDSv5
    az provider register --namespace Microsoft.Compute --wait
    az provider register --namespace Microsoft.ContainerService --wait
    az vm list-skus --location "$LOC" --size "$SIZE" --query "[?restrictions==\`[]\`].name" -o tsv
    az vm list-usage --location "$LOC" -o table | grep -i -E "Total Regional vCPUs|$FAMILY Family"

    az vm list-skus prints the size when your subscription can use it in this region. If it prints nothing, pick another region or another 4 vCPU, 32 GB size, such as Standard_E4s_v4 with the family ESv4.

    The two nodes need 8 vCPUs, in both rows of the Limit column: the regional total and the family of your size. New subscriptions often allow 10 vCPUs per region, and some free or credit-based offers allow only 4 and refuse increases. If your limit is too low, raise it, use another subscription, or create the cluster with --node-count 1 in step 2: one 4 vCPU, 32 GB node runs the full evaluation install. To raise a limit from the CLI:

    Raise a vCPU limit
    az extension add --name quota
    az provider register --namespace Microsoft.Quota --wait
    SCOPE="/subscriptions/$(az account show --query id -o tsv)/providers/Microsoft.Compute/locations/$LOC"
    az quota list --scope "$SCOPE" --query "[?name=='cores'].{name:name, limit:properties.limit.value}" -o table
    az quota create --resource-name cores --scope "$SCOPE" --limit-object value=8 --resource-type dedicated

    Repeat az quota create for the family; az quota list without the query shows its resource name, such as standardESv4Family. A request refused with ResourceNotAvailableForOffer means the subscription's offer does not allow increases.

  2. Create the cluster. Skip to the next step if you already have one. For a new evaluation cluster:

    Create the AKS cluster
    az group create --name "$RG" --location "$LOC" -o table
    az aks create --resource-group "$RG" --name "$AKS" --location "$LOC" \
      --node-count 2 --node-vm-size "$SIZE" --node-osdisk-size 100 \
      --tier free --generate-ssh-keys -o table 2>&1 | tee create-cluster.log
    az aks get-credentials --resource-group "$RG" --name "$AKS" --overwrite-existing
    chmod 600 ~/.kube/config
    kubectl get nodes
    kubectl get storageclass

    Creation takes a few minutes and prints one row with the cluster's state, Succeeded. --generate-ssh-keys creates a key pair in ~/.ssh if you have none. Each node shows Ready, and default shows (default) among the storage classes listed. Memory, not CPU, is what the install mostly needs; see Workloads and resources.

  3. Install an ingress controller and point DNS at it.

    traefik-values.yaml
    cat > traefik-values.yaml << 'EOF'
    ingressClass:
      enabled: true
      isDefaultClass: true
    EOF
    helm repo add traefik https://traefik.github.io/charts
    helm repo update
    helm install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml
    until LB_IP=$(kubectl -n traefik get svc traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}') \
      && [ -n "$LB_IP" ]; do sleep 10; done
    echo "$LB_IP"

    Create an A record from your UI hostname to that address.

    Check the path end to end
    curl -s -o /dev/null -w '%{http_code}\n' http://{ui_hostname}/

    It prints 404, from Traefik. Use traefik as INGRESS_CLASS in step 1 of the install.

  4. Set vm.max_map_count on every node. Apply the DaemonSet in Set vm.max_map_count with a DaemonSet.

  5. Issue certificates. Follow Issue certificates with cert-manager.

Image pulls on AKS. The release registry is on Amazon ECR, so the pull Secret from install step 4 is required on AKS. The server image is about 3 GB; on a cluster in westus2 the first pull takes about a minute and a half, and every node without the image repeats it. For production, mirror the images into a registry in your own region, such as Azure Container Registry; see Air-gapped installation.

Google GKE​

GKE needs the same preparation as AKS: an ingress controller and vm.max_map_count. Its Persistent Disk driver is installed with standard-rwo as the default StorageClass, and a load balancer Service receives a public IP address without annotations. Use a GKE Standard cluster. Autopilot clusters reject privileged pods, so the vm.max_map_count DaemonSet cannot run there and OpenSearch does not start. Run the steps from the machine you install from, with the Google Cloud CLI (gcloud) and its gke-gcloud-auth-plugin installed alongside the tools under Before you start.

Set the project variables
PROJECT={gcp_project_id}
GKE={cluster_name}
REGION={gcp_region}
ZONE={gcp_zone}
gcloud config set project "$PROJECT"
gcloud services enable container.googleapis.com compute.googleapis.com

  1. Check quota. New projects often allow 12 vCPUs across all regions, 8 for some machine families, and 250 GB of SSD-class disk per region.

    Check regional and project quotas
    gcloud compute regions describe "$REGION" --format=json \
      | jq -r '.quotas[] | select(.metric | IN("CPUS","E2_CPUS","SSD_TOTAL_GB","IN_USE_ADDRESSES")) | "\(.metric)\t\(.usage)\t\(.limit)"'
    gcloud compute project-info describe --format=json \
      | jq -r '.quotas[] | select(.metric=="CPUS_ALL_REGIONS") | "\(.metric)\t\(.usage)\t\(.limit)"'

    Each line shows the metric, current usage and limit. Default GKE boot disks and standard-rwo volumes both count against SSD_TOTAL_GB. Raise limits under IAM & Admin, then Quotas, in the Google Cloud console.

  2. Create the cluster. Skip to the next step if you already have one. For a new evaluation cluster:

    Create the GKE cluster
    gcloud container clusters create "$GKE" --zone "$ZONE" \
      --num-nodes 2 --machine-type e2-highmem-4 \
      --disk-type pd-standard --disk-size 50 \
      --release-channel regular 2>&1 | tee create-cluster.log
    kubectl get nodes
    kubectl get storageclass

    Creation takes about five minutes and switches your kubectl context to the new cluster. Two nodes show Ready, and standard-rwo shows (default). The cluster is zonal on purpose: Persistent Disk volumes are zonal, and a regional cluster also creates the requested node count in each of three zones. The boot disks use pd-standard so the SSD quota stays free for the PostgreSQL and OpenSearch volumes. Two e2-highmem-4 nodes (4 vCPUs and 32 GB each) run the evaluation install within an 8 vCPU quota; see Workloads and resources for what it needs.

  3. Install an ingress controller and point DNS at it. Follow Azure AKS step 3. The commands are identical: the Traefik Service receives a public IP address, which takes about a minute, and curl prints 404 once DNS points at it.

  4. Set vm.max_map_count on every node. Apply the DaemonSet in Set vm.max_map_count with a DaemonSet.

  5. Issue certificates. Follow Issue certificates with cert-manager.

Image pulls on GKE. The release registry is on Amazon ECR, so the pull Secret from install step 4 is required on GKE. The server image is about 3 GB; on the cluster above in us-west1 the first pull took about two and a half minutes, and every node without the image repeats it. For production, mirror the images into a registry in your own region, such as Artifact Registry; see Air-gapped installation.

Red Hat OpenShift​

OpenShift admits pods through security context constraints (SCCs), publishes Ingress objects through its own Router, and tunes nodes through the Node Tuning Operator, so it differs from the other platforms in four places: two SCC grants, a node tuning profile, Router-based ingress, and a pull Secret you supply yourself. The steps below were verified on Red Hat OpenShift Service on AWS (ROSA) with hosted control planes, OpenShift 4.22.

In this release the chart does not run under the default restricted-v2 SCC alone. A cluster administrator must grant the release's namespace the anyuid and nonroot-v2 SCCs. Every workload except OpenSearch is admitted under nonroot-v2, which allows any non-root user ID. OpenSearch is admitted under anyuid because its volume-ownership init container runs as root.

You need cluster-admin rights and the OpenShift CLI (oc). On ROSA you also need the ROSA CLI (rosa).

  1. Create the cluster. Skip to the next step if you already have one. For a ROSA evaluation cluster, first enable ROSA with hosted control planes on the ROSA page of the AWS console and link your Red Hat account there. Run the ROSA CLI as an IAM user, since it refuses root credentials.

    Set up ROSA roles and network
    CLUSTER={cluster_name}
    REGION={aws_region}
    rosa login --use-device-code
    rosa create ocm-role --mode auto --yes
    rosa create user-role --mode auto --yes
    rosa create account-roles --hosted-cp --mode auto --yes
    rosa create oidc-config --mode auto --yes
    rosa create network --param Region="$REGION" --param Name="$CLUSTER-net" \
      --param AvailabilityZoneCount=1 --param VpcCidr=10.0.0.0/16
    rosa list oidc-config

    rosa create network prints the new public and private subnet IDs at the end, and rosa list oidc-config prints the OIDC config ID. Set them as OIDC_ID and SUBNETS.

    Create the ROSA cluster
    OIDC_ID={oidc_config_id}
    SUBNETS={public_subnet_id},{private_subnet_id}
    rosa create cluster --cluster-name "$CLUSTER" --hosted-cp --sts --mode auto --yes \
      --region "$REGION" --oidc-config-id "$OIDC_ID" --subnet-ids "$SUBNETS" \
      --operator-roles-prefix "$CLUSTER" --compute-machine-type m6i.2xlarge --replicas 2
    rosa logs install -c "$CLUSTER" --watch
    rosa create admin -c "$CLUSTER"

    The cluster reports ready after about ten minutes, and its worker nodes follow about ten minutes later. rosa create admin prints an oc login command; the login takes a few minutes to become active. Run it.

    Wait for the cluster operators
    oc wait clusteroperators --all --for=condition=Available --timeout=30m
    oc get nodes

    Both workers show Ready. The network is in one Availability Zone for the same reason as on Amazon EKS: EBS volumes are zonal.

  2. Set vm.max_map_count on every node. OpenShift does not admit the privileged DaemonSet used on the other platforms, so set it through a tuning profile. On ROSA with hosted control planes, attach a tuning config to the worker machine pool:

    tuned-opensearch.yaml
    cat > tuned-opensearch.yaml << 'EOF'
    profile:
    - data: |
        [main]
        summary=Datastrato OpenSearch sysctl
        include=openshift-node
        [sysctl]
        vm.max_map_count=262144
      name: datastrato-opensearch
    recommend:
    - priority: 20
      profile: datastrato-opensearch
    EOF
    rosa create tuning-configs -c "$CLUSTER" --name datastrato-opensearch --spec-path tuned-opensearch.yaml
    rosa edit machinepool -c "$CLUSTER" workers --tuning-configs datastrato-opensearch

    On other OpenShift clusters, create the same profile as a Tuned resource in the openshift-cluster-node-tuning-operator namespace. Either way the setting applies to running nodes without replacing them.

    Check vm.max_map_count on every node
    for n in $(oc get nodes -o name); do
      echo -n "$n: "; oc debug "$n" -q -- chroot /host sysctl -n vm.max_map_count 2>/dev/null
    done

    Every node prints 262144.

  3. Create the namespace and grant the SCCs.

    Create the namespace and grant the SCCs
    oc new-project datastrato
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:datastrato
    oc adm policy add-scc-to-group nonroot-v2 system:serviceaccounts:datastrato

    The grants apply to every service account the chart creates in the namespace.

  4. Find your hostnames. The Router serves every hostname under the cluster's apps domain with a wildcard certificate, so no DNS record and no cert-manager are needed.

    Read the cluster apps domain
    oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}{"\n"}'

    Use gravitino. followed by that domain as UI_HOST in install step 1.

Changes to the install steps on OpenShift.

  • Step 1: set INGRESS_CLASS=openshift-default. TLS_SECRET is not used.

  • Step 4: the chart's registry-token renewal cannot run on OpenShift in this release, so create the pull Secret yourself instead of ecr-refresh-iam:

    Create the pull Secret by hand
    oc -n "$NAMESPACE" create secret docker-registry image-pull-secret \
      --docker-server="$REGISTRY" --docker-username=AWS \
      --docker-password="$(AWS_ACCESS_KEY_ID="$AKID" AWS_SECRET_ACCESS_KEY="$SKEY" \
        aws ecr get-login-password --region us-east-1)"
    unset AKID SKEY

    The token in that Secret expires after about twelve hours, after which new pulls fail. For anything beyond an evaluation, mirror the images into your own registry; see Air-gapped installation.

  • Step 5: in my-values.yaml, replace the global block with the one below, and in the ingress block replace the cert-manager annotation with the Router annotation and replace tls: and its entries with tls: []:

    my-values.yaml
    global:
      imagePullSecrets:
        - name: image-pull-secret
      pullSecrets:
        create: false
        name: image-pull-secret
        ecrRefresh:
          enabled: false
    my-values.yaml
    ingress:
      enabled: true
      className: "openshift-default"
      annotations:
        route.openshift.io/termination: edge
      # hosts: unchanged
      tls: []

    The Router turns each Ingress into an edge-terminated Route on the cluster certificate. Without the global.imagePullSecrets entry, the render in step 6 fails on the OpenSearch pull Secret check.

  • Step 9: there is no Certificate resource to check. oc get routes -n "$NAMESPACE" lists the Routes, each with edge termination.

  • Uninstall: run helm uninstall with --no-hooks. The chart's cleanup Job is not admitted on OpenShift.

Set vm.max_map_count with a DaemonSet​

On EKS, AKS and GKE, a DaemonSet applies the OpenSearch setting to each node, including nodes added later by scaling or replacement.

max-map-count.yaml
cat > max-map-count.yaml << 'EOF'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: max-map-count
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: max-map-count
  template:
    metadata:
      labels:
        app: max-map-count
    spec:
      initContainers:
        - name: sysctl
          image: public.ecr.aws/docker/library/busybox:1.36
          command: ["sysctl", "-w", "vm.max_map_count=262144"]
          securityContext:
            privileged: true
      containers:
        - name: hold
          image: public.ecr.aws/docker/library/busybox:1.36
          command: ["sleep", "infinity"]
EOF
kubectl apply -f max-map-count.yaml
kubectl -n kube-system rollout status ds/max-map-count
for p in $(kubectl -n kube-system get pods -l app=max-map-count -o name); do
  kubectl -n kube-system exec "$p" -c hold -- cat /proc/sys/vm/max_map_count
done

Every node prints 262144.

Issue certificates with cert-manager​

Skip this if you already have a TLS Secret for your hostname. Otherwise, install cert-manager and a Let's Encrypt issuer. The hostname must resolve to your ingress controller and port 80 must be reachable from the internet, since Let's Encrypt validates the hostname over it.

Install cert-manager and a Let's Encrypt issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace \
  --version v1.16.2 --set crds.enabled=true
kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s
cat > letsencrypt-issuer.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
EOF
kubectl apply -f letsencrypt-issuer.yaml
sleep 15
kubectl get clusterissuer letsencrypt

The issuer shows True in the READY column. If your ingress class is not traefik, change ingressClassName to match. The values file in install step 5 names this issuer under ingress.annotations, and cert-manager creates the Secret named by TLS_SECRET in install step 1.

During issuance, cert-manager runs one short-lived cm-acme-http-solver pod per hostname. They end in Error when cert-manager stops them after validation, which is expected.

Install​

  1. Set your variables. Every later command reads these, so this is the only place you type your own values. Keep this shell open for the whole procedure. If you open a new shell, run this block again first.

    Set the install variables
    VERSION={chart_version}
    REGISTRY=730335553010.dkr.ecr.us-east-1.amazonaws.com
    NAMESPACE=datastrato
    RELEASE=enterprise
    PKG={path_to_release_package}
    UI_HOST={ui_hostname}
    INGRESS_CLASS=traefik
    TLS_SECRET=gravitino-tls
    METALAKE={metalake_name}
    ADMIN_PASSWORD='{admin_password}'
    LICENSE_FILE=$(ls "$PKG"/*license-prod-*.txt)
    KEY_FILE=$(ls "$PKG"/*ecr-pull-user_accessKeys.csv)
    ls -l "$LICENSE_FILE" "$KEY_FILE"

    The ls lists one license file and one key file. If it reports a missing file, check PKG.

    VariableWhat to set
    VERSIONThe chart version of your release, as in the name of chart-release-summary-{version}.md, for example 1.3.0
    REGISTRYLeave as shown. It is the Datastrato release registry
    NAMESPACELeave as datastrato unless you have a reason to change it. On OpenShift it must match the namespace granted the SCCs
    RELEASELeave as enterprise. Later commands derive resource names from it
    PKGThe directory that holds the files from the release package
    UI_HOSTThe hostname users type into the browser. See below
    INGRESS_CLASStraefik if you followed Prepare the cluster; openshift-default on OpenShift; otherwise a name from kubectl get ingressclass
    TLS_SECRETThe Secret that holds the certificate for UI_HOST. With cert-manager, keep gravitino-tls; cert-manager creates it. With your own certificate, the name of the TLS Secret you created. Not used on OpenShift
    METALAKEThe name of the first metalake, which you create in step 10. The chart needs it at install time because the Iceberg REST service, the Lance REST service and the MCP server each serve one metalake
    ADMIN_PASSWORDThe password of the local admin account. 12 to 64 characters, and no quotes, backslashes or $, since it is written into a YAML file in step 5

    Choosing UI_HOST. It must resolve to your ingress controller, and with cert-manager it must also be reachable from the internet on port 80, since Let's Encrypt validates it there. Use a name in a domain you control, with the DNS record from your platform's step in Prepare the cluster. For an evaluation without a domain of your own, use sslip.io, a public DNS service that resolves any name containing an IP address to that address. Write the load balancer's public IP address with dashes: gravitino.203-0-113-10.sslip.io resolves to 203.0.113.10. On EKS the load balancer has a DNS name rather than an IP address; getent ahostsv4 "$LB_HOST" lists its addresses, which stay fixed for the life of the load balancer. On OpenShift, use gravitino. followed by the cluster's apps domain.

  2. Read the pull key. The CSV holds an access key ID and a secret access key, and the command below reads both into the shell without printing them.

    Read the pull key into the shell
    read -r AKID SKEY < <(awk -F',' 'NR==1{for(i=1;i<=NF;i++){h=tolower($i); gsub(/\r/,"",h); if(h ~ /access key id/)a=i; if(h ~ /secret access key/)s=i}} NR==2{gsub(/\r/,""); print $a, $s}' "$KEY_FILE")
    echo "key id: ${AKID:0:4}...  secret length: ${#SKEY}"

    The secret length should print 40. If either value is empty, check KEY_FILE from step 1.

  3. Log in to the registry and pull the chart. The registry username is always AWS. The password is a token exchanged from the pull key, valid for about twelve hours. When it expires, run the login again; the key itself does not expire.

    Log in to the registry and pull the chart
    AWS_ACCESS_KEY_ID="$AKID" AWS_SECRET_ACCESS_KEY="$SKEY" \
      aws ecr get-login-password --region us-east-1 \
      | helm registry login --username AWS --password-stdin "$REGISTRY"

    helm pull "oci://$REGISTRY/datastratocharts/enterprise-helm" --version "$VERSION"
    helm show values "./enterprise-helm-$VERSION.tgz" > "chart-defaults-$VERSION.yaml"
    ls -l "enterprise-helm-$VERSION.tgz"

    chart-defaults-$VERSION.yaml is the chart's full set of defaults, kept for reference. Do not edit it. Your settings go in a separate file in step 5, which holds only what you change, so a new chart version never forces you to merge.

  4. Create the namespace and the pull Secret. The cluster pulls images on its own schedule, on node restarts and rescheduling as well as at install, so it cannot rely on the token from step 3. The chart instead keeps the pull key in a Secret and renews the registry token itself, before the install and every six hours after.

    Create the namespace and pull Secret
    kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
    kubectl -n "$NAMESPACE" create secret generic ecr-refresh-iam \
      --from-literal=aws-access-key-id="$AKID" \
      --from-literal=aws-secret-access-key="$SKEY" \
      --dry-run=client -o yaml | kubectl apply -f -
    unset AKID SKEY
    kubectl -n "$NAMESPACE" get secret ecr-refresh-iam

  5. Write your values file. Run the block as-is. It substitutes your variables from step 1.

    my-values.yaml
    cat > my-values.yaml << EOF
    # my-values.yaml
    global:
      pullSecrets:
        create: false
        name: image-pull-secret
        ecrRefresh:
          enabled: true
          existingSecret: ecr-refresh-iam

    gravitino:
      authorization:
        enable: true
        serviceAdmins: admin
      authenticators: "basic"
      admin:
        initPassword: "$ADMIN_PASSWORD"
      icebergRest:
        dynamicConfigProvider:
          metalake: "$METALAKE"
      lanceRest:
        gravitinoMetalake: "$METALAKE"
      env:
        - name: GRAVITINO_MEM
          value: "-Xms4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:+ExitOnOutOfMemoryError"
      resources:
        requests:
          memory: 8Gi
        limits:
          memory: 8Gi

    mcp:
      enabled: true
      metalake: "$METALAKE"

    enterpriseWeb:
      env:
        - name: NEXTAUTH_URL
          value: "https://$UI_HOST"
        - name: AUTH_TRUST_HOST
          value: "true"

    ingress:
      enabled: true
      className: "$INGRESS_CLASS"
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt
      hosts:
        - host: $UI_HOST
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: '{{ include "enterprise.fullname" . }}-ui'
                  port:
                    number: 3000
            - path: /mcp
              pathType: Prefix
              backend:
                service:
                  name: '{{ include "enterprise.fullname" . }}-mcp'
                  port:
                    number: 8000
      tls:
        - secretName: $TLS_SECRET
          hosts:
            - $UI_HOST
    EOF
    grep -n -E 'metalake|host:|secretName|password|memory|cluster-issuer' my-values.yaml

    The grep shows your hostname, metalake, TLS Secret, password, memory and issuer in place. If any line shows an empty value, a variable from step 1 was not set.

    Run this step once. Running it again overwrites later edits. From here on, edit my-values.yaml directly.

    What each block does:

    SettingPurpose
    global.pullSecretsUses the Secret from step 4 and renews the registry token automatically
    gravitino.authenticators, gravitino.adminLocal password sign-in with an admin account. The account name comes from serviceAdmins, and admin.initPassword sets its password at the first install only; add your identity provider later, see Authentication
    gravitino.icebergRest.dynamicConfigProviderThe metalake the Iceberg REST service serves; the install is refused without it
    gravitino.lanceRestThe metalake the Lance REST service serves; the install is refused without it
    gravitino.env GRAVITINO_MEMServer heap and metaspace: a 4 GiB heap and 2 GiB of metaspace, twice the chart defaults, since metaspace grows with the number of catalogs and the features in use
    gravitino.resourcesMemory for the server pod. The chart refuses less than 7Gi with the heap and metaspace above; 8Gi leaves room for the JVM's own overhead
    mcp.metalakeThe metalake the MCP server serves
    enterpriseWeb.envThe address the UI is reached at. The chart creates the UI's session signing secret itself, in the Secret {release}-ui-auth, and keeps it across upgrades
    ingressPublishes the UI and the MCP server on your hostname
    ingress.annotationsHas cert-manager issue the certificate into the Secret named in tls, using the letsencrypt issuer from Issue certificates with cert-manager

    If you supply your own certificate in a TLS Secret instead of using cert-manager, delete the annotations block from ingress.

  6. Check the render. Rendering validates your values against the cluster without installing anything. Every configuration error the chart knows how to detect is reported here.

    Render the chart against the cluster
    helm template "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
      --dry-run=server > /tmp/render.yaml && echo "RENDER OK"

    It must print RENDER OK. Otherwise the message names the value that is wrong or missing; fix my-values.yaml and run it again.

    The license key is passed on the command line rather than stored in the values file, and tr strips the file's trailing newline, which the license check otherwise rejects as malformed.

  7. Install.

    Install the chart
    helm upgrade --install "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$LICENSE_FILE")" \
      --timeout 15m 2>&1 | tee helm-install.log

    Helm prints the release notes when it finishes. The pods are still starting at that point.

  8. Wait for the pods.

    Wait for the Gravitino rollout
    kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE"-gravitino --timeout=15m
    kubectl -n "$NAMESPACE" get pods

    When the rollout reports success, every pod should be Running with all containers ready, except the registry-token Job and the schema Job, which show Completed. Expect these workloads: the Gravitino server, its PostgreSQL, the UI, the MCP server, the metrics service, OpenSearch, Marquez, and Marquez web. The UI and MCP pods wait in Init until the Gravitino server is ready, and the metrics service sits at 0/1 for a while after starting; on a fresh cluster everything is ready within about four minutes. A schema Job pod may show Error once, if it starts before the database's name resolves, and be retried; that is normal when the Job shows Complete.

    Anything in ImagePullBackOff, CrashLoopBackOff or Init:Error has a cause listed under Troubleshoot.

  9. Check the certificate and the server.

    Check the certificate and the server
    kubectl -n "$NAMESPACE" get certificate 2>/dev/null
    kubectl -n "$NAMESPACE" port-forward svc/"$RELEASE"-gravitino 8090:8090 >/dev/null 2>&1 &
    sleep 3
    curl -s -u "admin:$ADMIN_PASSWORD" http://localhost:8090/api/metalakes | jq .

    If you use cert-manager, the certificate shows True in the READY column. The curl returns {"code": 0, "metalakes": []}. An empty list is correct, since the install creates no metalakes.

  10. Create the metalake. Use the same name you set as METALAKE in step 1.

    Create the metalake
    curl -s -u "admin:$ADMIN_PASSWORD" -X POST -H "Content-Type: application/json" \
      -d "{\"name\":\"$METALAKE\",\"comment\":\"\",\"properties\":{}}" \
      http://localhost:8090/api/metalakes | jq '{code, name: .metalake.name}'

    The response shows "code": 0 and your metalake name.

  11. Sign in to the UI. Open https://{ui_hostname} in a browser and sign in as admin with the password from step 1.

The install is complete. Keep the port forward running if you are continuing to Get started to connect a first catalog; otherwise stop it with kill %1.

Prepare for production​

The procedure above is the same one a production install follows. Production is not a separate install; it is the same my-values.yaml with the changes below, applied by running step 7 again. An evaluation can therefore grow into a production deployment without being reinstalled, apart from moving the metadata database, which is easiest to decide before you load real metadata.

AreaEvaluation, as installed aboveProduction
LicenseTrial key with a short expiryProduction key from your account team; replace it on the command line in step 7
Sign-inLocal admin account onlyYour identity provider, with oauth,basic so the local admin account remains for recovery; see Configure
Users and groupsCreated by handProvisioned from your directory through SCIM; see Configure
CredentialsAdmin password written into my-values.yamlHeld in Secrets you create and name in the values file, so it carries no credential; see Secrets
Metadata databasePostgreSQL deployed by the chartA database you operate and back up, with gravitino.postgresql.enabled: false; see Metadata database
ImagesPulled from the Datastrato release registryMirrored into your own registry with the offline toolkit, with global.registry pointed at it; see Air-gapped installation
Memory8Gi request and limit for the serverThe same, with requests.memory kept equal to limits.memory so the server is the last pod evicted under node pressure. Raise both if you raise the heap in GRAVITINO_MEM
BackupsNoneThe metadata database and lineage databases, the release values and the audit log; see Backup and restore
ReplicasOneOne, or several for query traffic; see Run more than one server

The TLS requirement is the same for both: a certificate the browser trusts. An evaluation on a self-signed certificate fails at sign-in, so it cannot be deferred to production.

Run more than one server​

Several server replicas can run behind the chart's Service, which spreads query traffic across them and keeps the IRC and the API answering while one pod restarts. Each replica keeps its own caches and drops its copies of anything changed through another replica within a few seconds, by checking a change log in the metadata database.

Several replicas suit query traffic:

  • Reading metadata through the IRC, the Gravitino REST API, the connectors and the UI, including loading tables, listing objects and scan planning.
  • Writing Iceberg tables. Each commit swaps the table's metadata pointer in the catalog backend in one step, so a commit that conflicts with one made through another replica is rejected with 409, as it is on a single server.
  • Creating and dropping Iceberg tables through the IRC. After each one, the server checks its own record of the table against the catalog backend and corrects it.

Make administrative changes one at a time: creating, changing or dropping metalakes, catalogs, schemas and Iceberg namespaces, and changes to users, groups, roles, grants, owners, tags and policies. Each server puts these changes in order with a lock held in its own memory, so two that arrive at the same moment through different replicas are not ordered against each other.

my-values.yaml
gravitino:
  replicas: 3
  logPersistence:
    accessModes:
      - ReadWriteMany
    storageClassName: {rwx_storage_class}

The chart refuses more than one replica on a ReadWriteOnce logs volume, so the logs need a storage class that supports ReadWriteMany, such as Amazon EFS, Azure Files, Google Filestore or NFS. Each replica writes to a directory of its own on it. A JDBC driver volume needs ReadWriteMany for the same reason. Run the metadata database outside the cluster, as in High availability, and check that your license key allows the number of replicas; see Licensing.

Configure​

Every change is made in my-values.yaml and applied by running step 7 again with the same command. Keep the license on the command line each time.

To do thisSet
Sign in through your identity providergravitino.authenticators: "oauth,basic", gravitino.authenticator.oauth.*, and gravitino.browserSso; see Authentication
Publish the Gravitino REST API to clients outside the clusteringress.api.enabled, ingress.api.host, and ingress.api.tls; see Publish the REST API
Publish the Iceberg REST service to enginesingress.iceberg.enabled, ingress.iceberg.host, and ingress.iceberg.tls
Set the Iceberg REST default cataloggravitino.icebergRest.dynamicConfigProvider.defaultCatalogName
Provision users and groups from a directorygravitino.scim.enabled, ingress.scim.enabled and ingress.scim.host; see SCIM provisioning
Publish the Lance REST service to clientsingress.lance.enabled, ingress.lance.host, and ingress.lance.tls
Publish the MCP server on a hostname of its owningress.mcp.enabled and ingress.mcp.host, with the /mcp path removed from ingress.hosts
Preview table data through your own Trinogravitino.preview.enabled and gravitino.preview.jdbcUrl
Supply JDBC drivers that cannot ship in the image, such as MySQLgravitino.jdbcDrivers.enabled and gravitino.jdbcDrivers.mode
Use your own metadata databaseSee Metadata database
Expose the server without an ingress controllergravitino.service.type: LoadBalancer; see Expose a Service directly

The chart deploys PostgreSQL for the metadata database by default. Its data sits on a persistent volume that survives upgrades and uninstalls. For production, a database you operate and back up is the better choice.

The admin password is set once, at first install. The chart keeps it in a Secret that survives upgrades, so changing gravitino.admin.initPassword later has no effect; change the password through the UI or the API instead.

gravitino.env replaces the chart's environment list rather than adding to it. When you add a variable, keep the GRAVITINO_MEM entry in the same list.

Publish the REST API​

Clients outside the cluster, such as the Java and Python SDKs and the Spark, Trino and Flink connectors, call the Gravitino REST API. Publish it on a hostname of its own with ingress.api. The UI hostname also answers /api, but those calls pass through the UI pod, which forwards them for the browser's signed-in session. Under load from clients the UI pod limits throughput, and calls fail while it restarts. The chart rejects an ingress.api.host equal to the UI hostname.

my-values.yaml
ingress:
  api:
    enabled: true
    host: {api_hostname}
    tls:
      - secretName: api-tls
        hosts:
          - {api_hostname}

Add these under the existing ingress block in my-values.yaml rather than as a second ingress key, and run step 7 again. Create a DNS record for {api_hostname} pointing at the same ingress controller as the UI hostname. With cert-manager, the issuer annotation on ingress covers the new host and cert-manager creates api-tls.

Point every client outside the cluster at https://{api_hostname}. The SDKs and connectors append /api themselves, so give the address without it:

ClientSetting
Java SDKGravitinoClient.builder(uri)
Python SDKGravitinoClient(uri=...)
Spark connectorspark.sql.gravitino.uri
Trino connectorgravitino.uri
Flink connectortable.catalog-store.gravitino.gravitino.uri

Check that the hostname reaches the server:

Check the REST API hostname
curl -s -o /dev/null -w '%{http_code}\n' https://{api_hostname}/api/version

It prints 200, or 401 when the call needs credentials.

MCP server​

The mcp block configures the MCP server. The chart deploys it by default, and the top-level ingress publishes it at /mcp on the UI hostname. To publish it on a hostname of its own instead, use ingress.mcp; the chart rejects mcp.ingress.

SettingDescriptionDefault
mcp.enabledDeploy the MCP servertrue
mcp.metalakeMetalake a tool call uses when it names none. Leave empty to have agents name the metalake on every call(none)
mcp.gravitinoUriGravitino server the MCP server calls. Leave empty to use the Gravitino server in the same release(same release)
mcp.extraArgsAdditional server arguments, such as ["--include-tool-tags", "catalog,schema,table"]; see MCP tools[]
mcp.auth.token, mcp.auth.existingSecretStatic Bearer token for the service identity(none)
mcp.auth.oauth.*OAuth client credentials for the service identity(none)
mcp.auth.oauth.noServiceIdentityFallbackReject requests that carry no Authorization header instead of using the service identityfalse
mcp.tls.enabled, mcp.tls.existingSecretServe HTTPS from the pod itself, from a kubernetes.io/tls Secret. Not needed when the ingress terminates TLSfalse
mcp.replicaCountNumber of replicas. The server keeps no state, so replicas scale freely1
mcp.extraVolumes, mcp.extraVolumeMountsAdditional volumes, such as one that keeps the audit log across restarts[]

Each agent normally sends its own user's credentials, and the server forwards them to Gravitino. A service identity covers requests that arrive without an Authorization header, and every such request then acts as that one identity. There are two ways to provide it:

  • A static token, in mcp.auth.token or in the Secret named by mcp.auth.existingSecret. It is always sent as a Bearer token, so it must be an OAuth access token, and it stops working when that token expires.
  • OAuth client credentials, in mcp.auth.oauth.tokenEndpoint, clientId, and clientSecret (or existingSecret), which must be set together. The server fetches its own token and refreshes it, so a long-running server does not start failing when a token expires.

A static token wins when both are set. Set mcp.auth.oauth.noServiceIdentityFallback to true to keep the client credentials while still rejecting requests that carry no identity of their own.

Set other server properties​

Set any Gravitino server property that has no dedicated chart value under gravitino.additionalConfigItems in my-values.yaml, and apply it by running step 7 again. The chart writes each entry into the server's gravitino.conf as key = value:

my-values.yaml
gravitino:
  additionalConfigItems:
    gravitino.cache.lockSegments: "16"

Every value must be a quoted string in my-values.yaml; the chart fails to render an unquoted number or boolean. To set a value on the command line, use --set-string rather than --set, and escape the dots in the property name: --set-string 'gravitino.additionalConfigItems.gravitino\.cache\.lockSegments=16'.

Use a property's dedicated chart value when one exists, and use additionalConfigItems only for properties without one, because setting a property both ways writes it twice into gravitino.conf.

Google Cloud credentials​

Catalogs on Google Cloud Storage or BigQuery read a service account key from the pod. Create the Secret before installing or upgrading, since the chart refuses to render a mount for a Secret that does not exist.

Create the service account key Secret
kubectl -n "$NAMESPACE" create secret generic gcs-service-account \
  --from-file=key.json={path_to_service_account_key}
my-values.yaml
gravitino:
  env:
    - name: GRAVITINO_MEM
      value: "-Xms4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:+ExitOnOutOfMemoryError"
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /etc/gcs/key.json
  extraVolumes:
    - name: gcs-credentials
      secret:
        secretName: gcs-service-account
        items:
          - key: key.json
            path: key.json
  extraVolumeMounts:
    - name: gcs-credentials
      mountPath: /etc/gcs
      readOnly: true

S3 and Azure credentials are catalog properties rather than pod configuration. See Object storage.

Air-gapped installation​

The offline toolkit in the release package copies every image the chart uses into your own registry. Its README covers the download, upload and push modes. Push the chart to the same registry, then set global.registry in my-values.yaml to redirect every image there. With images in your own registry, global.pullSecrets.ecrRefresh and the ecr-refresh-iam Secret are no longer needed; supply whatever pull credential your own registry requires instead.

Upgrade​

To move to a new version, follow Upgrades. It backs up the metadata database and records what the server holds before running helm upgrade with your existing my-values.yaml, and it covers rolling back.

Uninstall​

Uninstall the release
helm uninstall "$RELEASE" -n "$NAMESPACE"

Uninstalling leaves the persistent volumes of the built-in PostgreSQL and OpenSearch in place, along with the admin credentials and database password Secrets (see Secrets), so a reinstall under the same release name picks up the existing data. The logs volume is deleted, and the audit log on it; copy or ship the audit log first, as described in Backup and restore. For a clean slate, delete the namespace as well.

Delete the namespace
kubectl delete namespace "$NAMESPACE"

Nothing in your connected sources is affected either way, since Gravitino never held that data.

Removing an evaluation cluster. On EKS, AKS and GKE, delete the namespace and wait for its volumes to go before you delete the cluster. Deleting the cluster first can leave the PostgreSQL and OpenSearch cloud disks behind, still billed, and on EKS can also leave the ingress load balancer. Delete the ingress controller release too, so its load balancer is removed while the cluster can still clean it up.

Remove the namespace and ingress controller
kubectl delete namespace "$NAMESPACE" --timeout=5m
helm uninstall traefik -n traefik
kubectl get pv

kubectl get pv prints No resources found once every volume is deleted. Then delete the cluster with your platform's command (eksctl delete cluster, az group delete, gcloud container clusters delete, or rosa delete cluster). On ROSA, also delete the operator roles, the OIDC config and the network stack afterwards, since the network's NAT gateway is billed by the hour. On Azure, a NetworkWatcherRG resource group may remain after az group delete. Azure creates it automatically with the cluster's network; it costs nothing and can stay.

Troubleshoot​

SymptomCause and fix
Render fails saying a license key is requiredThe --set-string gravitino.license.key argument is missing or LICENSE_FILE is empty. Run the step 1 block again and check its ls output
Render fails on the Gravitino server's memorygravitino.resources is missing or below 7Gi. Set the memory request and limit as in step 5
Render fails naming dynamicConfigProvider.metalakegravitino.icebergRest.dynamicConfigProvider.metalake is unset in my-values.yaml
Render fails naming lanceRest.gravitinoMetalakegravitino.lanceRest.gravitinoMetalake is unset in my-values.yaml. Set it to the same metalake as METALAKE, as in step 5
Render fails with admin.username was removed or admin.password was renamed to admin.initPasswordThe values file was written for an earlier release. Delete gravitino.admin.username and rename gravitino.admin.password to gravitino.admin.initPassword, as in step 5
Render fails with Do not set AUTH_SECRET or NEXTAUTH_SECRET in enterpriseWeb.envDelete that entry from enterpriseWeb.env. The chart creates the session secret itself; to supply your own, set enterpriseWeb.authSecret.existingSecret
Render fails with gravitino.scim.ingress.enabled is not supportedUse ingress.scim.enabled and ingress.scim.host instead, as in Configure
Render fails naming a Secret and giving a kubectl create secret commandA Secret referenced by extraVolumes does not exist yet. Run the command it prints
Server logs report INVALID_FORMAT for the licenseThe key was passed with its trailing newline. Pass it through tr -d '\n' as in step 7
Pods in ImagePullBackOffThe ecr-refresh-iam Secret is missing or holds the wrong key. Repeat step 4, then delete the failing pods
OpenSearch restartingvm.max_map_count is below 262144 on its node
Sign-in returns to the login page, or shows a configuration errorThe browser does not trust the certificate, or NEXTAUTH_URL does not match the address in the browser
Gravitino pod restarts after running for a day or moreMetaspace exhausted. Raise -XX:MaxMetaspaceSize in GRAVITINO_MEM above the 2g set in step 5, and raise the memory request and limit by the same amount. The chart's own default, used when GRAVITINO_MEM is not set, is 1g
After a reinstall, old catalogs and users are back, or the admin password in the values file is rejectedThe release reused the kept PostgreSQL volume, which holds the old metadata and the old admin password. Sign in with the old password, or delete the namespace for a clean start, as described under Uninstall
The server pod stays in Init and then failsThe schema Job could not reach or prepare the metadata database. Read its log; see Metadata database
The server pod stays Pending with Insufficient memoryNo node has 8Gi free for the server's memory request. Add a larger node, or free memory on an existing one
curl in step 9 returns connection refusedThe port forward has stopped. Run the port-forward line again
PersistentVolumeClaims stay Pending on EKSThe EBS CSI driver add-on is missing, or no StorageClass is marked default. See Amazon EKS steps 2 and 3
A pod stays Pending with a volume node affinity conflict on EKSNo node is running in the Availability Zone of its EBS volume. Add a node group in that zone
Certificate never becomes readyLet's Encrypt cannot reach port 80 on the hostname. Check that DNS points at the ingress controller and port 80 is open
az aks create fails with a quota errorThe subscription's vCPU quota is too low for the node size and count. See Azure AKS step 1
OpenSearch never starts on GKE and the max-map-count DaemonSet is rejectedThe cluster is GKE Autopilot, which rejects privileged pods. Use a GKE Standard cluster
On OpenShift, a Job or pod is never created and events show unable to validate against any security context constraintThe namespace lacks the SCC grants. See Red Hat OpenShift step 3
On OpenShift, the registry-token Job's ecr-get-login-password container shows container has runAsNonRoot and image will run as rootRegistry-token renewal is enabled. Disable it and supply the pull Secret as described under Red Hat OpenShift
Render fails with opensearch.imagePullSecrets lists "image-pull-secret", but this release will not create that SecretRegistry-token renewal is disabled and the Secret is supplied by hand. Add image-pull-secret under global.imagePullSecrets
kubectl or helm fails with tls: bad record MACPackets are corrupted on the network path from your machine, usually an MTU mismatch behind a VPN or under WSL. Lower the interface MTU, for example sudo ip link set dev eth0 mtu 1350, and retry
Last updated on