Skip to main content

Install with Helm

Datastrato Enterprise installs from a Helm chart. The chart and every image it deploys are served from the Datastrato release registry on Amazon ECR, and one pull-only credential covers both. Follow the steps in order. Each one ends with something you can check before moving on, so a problem shows up at the step that caused it rather than three steps later.

The install steps are the same on every Kubernetes cluster. What differs is how the cluster meets the requirements under Before You Start. Prepare the Cluster covers the five tested platforms: k3s on your own hosts, Amazon EKS, Azure AKS, Google GKE, and Red Hat OpenShift.

Expect thirty minutes, most of it waiting for images to pull on the first install.

Before you start​

You need these tools on the machine you install from: kubectl pointed at the target cluster, helm 3, curl, jq, openssl, and the AWS CLI (aws). The AWS CLI is needed on every platform, not only on AWS, because the Datastrato release registry is on Amazon ECR and step 3 uses it to turn the pull key into a registry login. An air-gapped install that pulls from your own registry does not need it.

You need these files from Datastrato. Your account team sends a release package as a zip file wrapping a tarball. Unpack both layers, then note the directory that holds the files below; step 1 calls it PKG.

Unpack the release package
unzip gravitino-enterprise-{version}-artifacts.zip
tar -xzf gravitino-enterprise-{version}-artifacts.tar.gz
find . -name gravitino-ecr-pull-user_accessKeys.csv

The package contains:

FileWhat it is
gravitino-license-prod-{date}.txtYour license key
gravitino-ecr-pull-user_accessKeys.csvA pull-only AWS access key for the release registry; keep private
offline-toolkit-{version}.tar.gzImage mirroring tool, needed only for an air-gapped install
chart-release-summary-{version}.mdWhat changed in this release

Your cluster needs these. See Requirements for sizing.

  • vm.max_map_count set to at least 262144 on every node. OpenSearch will not start without it.
  • An ingress controller, and its class name (for example traefik or nginx). Find it with kubectl get ingressclass.
  • A DNS hostname that resolves to that ingress controller, for the UI.
  • A TLS certificate for that hostname, either as a Kubernetes TLS Secret or issued by cert-manager. Use a certificate the browser trusts. With a self-signed certificate the browser discards the sign-in cookies and login fails in a way that looks like a credential problem.
  • A default StorageClass, since the built-in PostgreSQL and OpenSearch each claim a persistent volume. Check with kubectl get storageclass.
  • Enough memory for the Gravitino pod to hold a 4 GiB heap and 2 GiB of metaspace, plus the other workloads. Run one replica. Multiple replicas are not supported in this release.

Prepare the Cluster shows how to provide each of these on k3s, Amazon EKS, Azure AKS, Google GKE and Red Hat OpenShift.

Prepare the cluster​

Follow the subsection for your platform, then continue with Install. Both platforms issue certificates the same way, described in Issue Certificates with cert-manager.

k3s on your own hosts​

k3s supplies two of the requirements itself: Traefik as the ingress controller, with class name traefik, listening on ports 80 and 443 of each node, and the local-path StorageClass as the default.

Install k3s on the first node and give your user a copy of its kubeconfig. k3s writes the original as readable by root only.

Install k3s and copy the kubeconfig
curl -sfL https://get.k3s.io | sh -
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown "$USER" ~/.kube/config
kubectl get ingressclass
kubectl get storageclass

The first list shows traefik and the second shows local-path (default).

Set vm.max_map_count on every node, and persist it so it survives a reboot.

Set vm.max_map_count persistently
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee /etc/sysctl.d/99-opensearch.conf

Point DNS at the node. Create an A record from your UI hostname to the public IP address of the node, or of a load balancer in front of the nodes. Allow inbound ports 80 and 443 in the host firewall or security group. Port 80 must stay open if cert-manager issues the certificate, since Let's Encrypt validates the hostname over it.

local-path volumes live on one node's disk, so the PostgreSQL and OpenSearch pods can only run on the node that holds their data. On a multi-node k3s cluster, that node is a single point of failure for the entity store unless you use your own database; see Prepare for Production.

Amazon EKS​

An EKS cluster provides the control plane and the nodes, and none of the other requirements. It has no ingress controller, no working default StorageClass until the EBS CSI driver is added, and its managed nodes take no sysctl changes by hand. The steps below supply each one. Run them from the machine you install from, with eksctl installed alongside the tools under Before You Start.

Set the cluster variables
CLUSTER={cluster_name}
REGION={aws_region}
AZ_NODES={availability_zone_for_nodes}
AZ_SECOND={second_availability_zone}

  1. Create the cluster. Skip to the next step if you already have one. In that case, confirm it has the EBS CSI driver add-on:

    Check the EBS CSI driver add-on
    aws eks describe-addon --cluster-name "$CLUSTER" --region "$REGION" --addon-name aws-ebs-csi-driver \
      --query 'addon.status' --output text

    It must print ACTIVE. If the add-on is missing, install it with its IAM role as described in the Amazon EKS documentation for the Amazon EBS CSI driver.

    For a new evaluation cluster:

    cluster.yaml
    cat > cluster.yaml << EOF
    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    metadata:
      name: $CLUSTER
      region: $REGION
    availabilityZones: [$AZ_NODES, $AZ_SECOND]
    iam:
      withOIDC: true
    addons:
      - name: vpc-cni
      - name: coredns
      - name: kube-proxy
      - name: eks-pod-identity-agent
      - name: aws-ebs-csi-driver
        wellKnownPolicies:
          ebsCSIController: true
    managedNodeGroups:
      - name: ng-1
        instanceType: m6i.2xlarge
        availabilityZones: [$AZ_NODES]
        desiredCapacity: 3
        minSize: 3
        maxSize: 4
        volumeSize: 100
    EOF
    eksctl create cluster -f cluster.yaml 2>&1 | tee create-cluster.log
    kubectl get nodes

    Creation takes about twenty minutes. eksctl switches your kubectl context to the new cluster, and kubectl get nodes shows three nodes Ready.

    The node group sits in a single Availability Zone on purpose. EBS volumes are zonal: the PostgreSQL and OpenSearch volumes are created in one zone, and their pods can run only on nodes in that zone. A node group spread across zones can reschedule those pods where their volumes cannot attach. If you run nodes in several zones, keep a node group in every zone that holds a volume. Three m6i.2xlarge nodes run the evaluation install with room to spare; see Requirements for sizing.

  2. Make gp3 the default StorageClass.

    gp3.yaml
    cat > gp3.yaml << 'EOF'
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: gp3
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: ebs.csi.aws.com
    parameters:
      type: gp3
    volumeBindingMode: WaitForFirstConsumer
    allowVolumeExpansion: true
    EOF
    kubectl apply -f gp3.yaml
    kubectl annotate storageclass gp2 storageclass.kubernetes.io/is-default-class=false --overwrite 2>/dev/null
    kubectl get storageclass

    Only gp3 shows (default).

  3. Install an ingress controller. The example runs Traefik behind a Network Load Balancer. An NLB keeps a fixed address in each zone for its lifetime. Cross-zone load balancing lets the address in every zone reach the nodes, which here sit in one zone.

    traefik-values.yaml
    cat > traefik-values.yaml << 'EOF'
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    ingressClass:
      enabled: true
      isDefaultClass: true
    EOF
    helm repo add traefik https://traefik.github.io/charts
    helm repo update
    helm install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml
    kubectl -n traefik get pods

    The Traefik pod shows 1/1 Running. Use traefik as INGRESS_CLASS in step 1 of the install.

  4. Point DNS at the load balancer.

    Read the load balancer hostname
    until LB_HOST=$(kubectl -n traefik get svc traefik -o jsonpath='{.status.loadBalancer.ingress[0].hostname}') \
      && [ -n "$LB_HOST" ]; do sleep 10; done
    echo "$LB_HOST"

    Create a CNAME record, or a Route 53 alias record, from your UI hostname to that name. A new NLB name takes a few minutes to resolve.

    Check the path end to end
    curl -s -o /dev/null -w '%{http_code}\n' http://{ui_hostname}/

    It prints 404. The answer comes from Traefik, which is reachable and has no routes yet.

  5. Set vm.max_map_count on every node. A DaemonSet applies the setting to each node, including nodes added later by scaling or replacement.

    max-map-count.yaml
    cat > max-map-count.yaml << 'EOF'
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: max-map-count
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          app: max-map-count
      template:
        metadata:
          labels:
            app: max-map-count
        spec:
          initContainers:
            - name: sysctl
              image: busybox:1.36
              command: ["sysctl", "-w", "vm.max_map_count=262144"]
              securityContext:
                privileged: true
          containers:
            - name: hold
              image: busybox:1.36
              command: ["sleep", "infinity"]
    EOF
    kubectl apply -f max-map-count.yaml
    kubectl -n kube-system rollout status ds/max-map-count
    for p in $(kubectl -n kube-system get pods -l app=max-map-count -o name); do
      kubectl -n kube-system exec "$p" -c hold -- cat /proc/sys/vm/max_map_count
    done

    Every node prints 262144.

Image pulls on EKS. The node IAM role cannot pull from the Datastrato release registry, so the pull Secret from install step 4 is required on EKS exactly as on any other cluster. The release registry is in ap-northeast-1 and the server image is about 3 GB. On a cluster in us-east-2 the first pull takes about two minutes, and every node without the image repeats it, which on EKS includes node replacement, managed node group upgrades and scale-out. For production on EKS, mirror the images into a registry in your own region; see Air-Gapped Installation.

Azure AKS​

AKS needs less preparation than EKS. Its Azure Disk driver is installed with a default StorageClass already set, and a load balancer Service receives a plain public IP address, so no load balancer annotations are needed. What remains is the ingress controller and vm.max_map_count. Run the steps from the machine you install from, with the Azure CLI (az) installed alongside the tools under Before You Start.

Set the cluster variables
RG={resource_group}
AKS={cluster_name}
LOC={azure_region}

  1. Check vCPU quota. New subscriptions often allow 10 vCPUs per region and none for many VM families, and a cluster that exceeds the quota fails to create.

    Register providers and check vCPU quota
    az provider register --namespace Microsoft.Compute --wait
    az provider register --namespace Microsoft.ContainerService --wait
    az vm list-usage --location "$LOC" -o table | grep -i -E 'Total Regional vCPUs|DSv5|EBDSv5'

    The Limit column must cover the nodes you plan to create, both in the regional total and in the family of your VM size. Raise it under Subscriptions, then Usage + quotas, in the Azure portal.

  2. Create the cluster. Skip to the next step if you already have one. For a new evaluation cluster:

    Create the AKS cluster
    az group create --name "$RG" --location "$LOC" -o table
    az aks create --resource-group "$RG" --name "$AKS" --location "$LOC" \
      --node-count 2 --node-vm-size Standard_E4bds_v5 --node-osdisk-size 100 \
      --tier free --generate-ssh-keys 2>&1 | tee create-cluster.log | grep -v '^ '
    az aks get-credentials --resource-group "$RG" --name "$AKS" --overwrite-existing
    chmod 600 ~/.kube/config
    kubectl get nodes
    kubectl get storageclass

    Creation takes about five minutes. Two nodes show Ready, and default shows (default). Two Standard_E4bds_v5 nodes (4 vCPUs and 32 GB each) run the evaluation install within a 10 vCPU quota; memory, not CPU, is what the install mostly needs. See Requirements for sizing.

  3. Install an ingress controller and point DNS at it.

    traefik-values.yaml
    cat > traefik-values.yaml << 'EOF'
    ingressClass:
      enabled: true
      isDefaultClass: true
    EOF
    helm repo add traefik https://traefik.github.io/charts
    helm repo update
    helm install traefik traefik/traefik -n traefik --create-namespace -f traefik-values.yaml
    until LB_IP=$(kubectl -n traefik get svc traefik -o jsonpath='{.status.loadBalancer.ingress[0].ip}') \
      && [ -n "$LB_IP" ]; do sleep 10; done
    echo "$LB_IP"

    Create an A record from your UI hostname to that address.

    Check the path end to end
    curl -s -o /dev/null -w '%{http_code}\n' http://{ui_hostname}/

    It prints 404, from Traefik. Use traefik as INGRESS_CLASS in step 1 of the install.

  4. Set vm.max_map_count on every node. Apply the same DaemonSet as in Amazon EKS step 5 and run its check. Every node prints 262144.

Image pulls on AKS. The release registry is on Amazon ECR, so the pull Secret from install step 4 is required on AKS. The server image is about 3 GB; on a cluster in westus2 the first pull takes about a minute and a half, and every node without the image repeats it. For production, mirror the images into a registry in your own region, such as Azure Container Registry; see Air-Gapped Installation.

Google GKE​

GKE needs the same preparation as AKS: an ingress controller and vm.max_map_count. Its Persistent Disk driver is installed with standard-rwo as the default StorageClass, and a load balancer Service receives a public IP address without annotations. Use a GKE Standard cluster. Autopilot clusters reject privileged pods, so the vm.max_map_count DaemonSet cannot run there and OpenSearch does not start. Run the steps from the machine you install from, with the Google Cloud CLI (gcloud) and its gke-gcloud-auth-plugin installed alongside the tools under Before You Start.

Set the project variables
PROJECT={gcp_project_id}
GKE={cluster_name}
REGION={gcp_region}
ZONE={gcp_zone}
gcloud config set project "$PROJECT"
gcloud services enable container.googleapis.com compute.googleapis.com

  1. Check quota. New projects often allow 12 vCPUs across all regions, 8 for some machine families, and 250 GB of SSD-class disk per region.

    Check regional and project quotas
    gcloud compute regions describe "$REGION" --format=json \
      | jq -r '.quotas[] | select(.metric | IN("CPUS","E2_CPUS","SSD_TOTAL_GB","IN_USE_ADDRESSES")) | "\(.metric)\t\(.usage)\t\(.limit)"'
    gcloud compute project-info describe --format=json \
      | jq -r '.quotas[] | select(.metric=="CPUS_ALL_REGIONS") | "\(.metric)\t\(.usage)\t\(.limit)"'

    Each line shows the metric, current usage and limit. Default GKE boot disks and standard-rwo volumes both count against SSD_TOTAL_GB. Raise limits under IAM & Admin, then Quotas, in the Google Cloud console.

  2. Create the cluster. Skip to the next step if you already have one. For a new evaluation cluster:

    Create the GKE cluster
    gcloud container clusters create "$GKE" --zone "$ZONE" \
      --num-nodes 2 --machine-type e2-highmem-4 \
      --disk-type pd-standard --disk-size 50 \
      --release-channel regular 2>&1 | tee create-cluster.log
    kubectl get nodes
    kubectl get storageclass

    Creation takes about five minutes and switches your kubectl context to the new cluster. Two nodes show Ready, and standard-rwo shows (default). The cluster is zonal on purpose: Persistent Disk volumes are zonal, and a regional cluster also creates the requested node count in each of three zones. The boot disks use pd-standard so the SSD quota stays free for the PostgreSQL and OpenSearch volumes. Two e2-highmem-4 nodes (4 vCPUs and 32 GB each) run the evaluation install within an 8 vCPU quota; see Requirements for sizing.

  3. Install an ingress controller and point DNS at it. Follow Azure AKS step 3. The commands are identical: the Traefik Service receives a public IP address, which takes about a minute, and curl prints 404 once DNS points at it.

  4. Set vm.max_map_count on every node. Apply the same DaemonSet as in Amazon EKS step 5 and run its check. Every node prints 262144.

Image pulls on GKE. The release registry is on Amazon ECR, so the pull Secret from install step 4 is required on GKE. The server image is about 3 GB; on the cluster above in us-west1 the first pull took about two and a half minutes, and every node without the image repeats it. For production, mirror the images into a registry in your own region, such as Artifact Registry; see Air-Gapped Installation.

Red hat OpenShift​

OpenShift admits pods through security context constraints (SCCs), publishes Ingress objects through its own Router, and tunes nodes through the Node Tuning Operator, so it differs from the other platforms in four places: two SCC grants, a node tuning profile, Router-based ingress, and a pull Secret you supply yourself. The steps below were verified on Red Hat OpenShift Service on AWS (ROSA) with hosted control planes, OpenShift 4.22.

In this release the chart does not run under the default restricted-v2 SCC alone. A cluster administrator must grant the release's namespace the anyuid and nonroot-v2 SCCs. Every workload except OpenSearch is admitted under nonroot-v2, which allows any non-root user ID. OpenSearch is admitted under anyuid because its volume-ownership init container runs as root.

You need cluster-admin rights and the OpenShift CLI (oc). On ROSA you also need the ROSA CLI (rosa).

  1. Create the cluster. Skip to the next step if you already have one. For a ROSA evaluation cluster, first enable ROSA with hosted control planes on the ROSA page of the AWS console and link your Red Hat account there. Run the ROSA CLI as an IAM user, since it refuses root credentials.

    Set up ROSA roles and network
    CLUSTER={cluster_name}
    REGION={aws_region}
    rosa login --use-device-code
    rosa create ocm-role --mode auto --yes
    rosa create user-role --mode auto --yes
    rosa create account-roles --hosted-cp --mode auto --yes
    rosa create oidc-config --mode auto --yes
    rosa create network --param Region="$REGION" --param Name="$CLUSTER-net" \
      --param AvailabilityZoneCount=1 --param VpcCidr=10.0.0.0/16
    rosa list oidc-config

    rosa create network prints the new public and private subnet IDs at the end, and rosa list oidc-config prints the OIDC config ID. Set them as OIDC_ID and SUBNETS.

    Create the ROSA cluster
    OIDC_ID={oidc_config_id}
    SUBNETS={public_subnet_id},{private_subnet_id}
    rosa create cluster --cluster-name "$CLUSTER" --hosted-cp --sts --mode auto --yes \
      --region "$REGION" --oidc-config-id "$OIDC_ID" --subnet-ids "$SUBNETS" \
      --operator-roles-prefix "$CLUSTER" --compute-machine-type m6i.2xlarge --replicas 2
    rosa logs install -c "$CLUSTER" --watch
    rosa create admin -c "$CLUSTER"

    The cluster reports ready after about ten minutes, and its worker nodes follow about ten minutes later. rosa create admin prints an oc login command; the login takes a few minutes to become active. Run it.

    Wait for the cluster operators
    oc wait clusteroperators --all --for=condition=Available --timeout=30m
    oc get nodes

    Both workers show Ready. The network is in one Availability Zone for the same reason as on Amazon EKS: EBS volumes are zonal.

  2. Set vm.max_map_count on every node. OpenShift does not admit the privileged DaemonSet used on the other platforms, so set it through a tuning profile. On ROSA with hosted control planes, attach a tuning config to the worker machine pool:

    tuned-opensearch.yaml
    cat > tuned-opensearch.yaml << 'EOF'
    profile:
    - data: |
        [main]
        summary=Datastrato OpenSearch sysctl
        include=openshift-node
        [sysctl]
        vm.max_map_count=262144
      name: datastrato-opensearch
    recommend:
    - priority: 20
      profile: datastrato-opensearch
    EOF
    rosa create tuning-configs -c "$CLUSTER" --name datastrato-opensearch --spec-path tuned-opensearch.yaml
    rosa edit machinepool -c "$CLUSTER" workers --tuning-configs datastrato-opensearch

    On other OpenShift clusters, create the same profile as a Tuned resource in the openshift-cluster-node-tuning-operator namespace. Either way the setting applies to running nodes without replacing them.

    Check vm.max_map_count on every node
    for n in $(oc get nodes -o name); do
      echo -n "$n: "; oc debug "$n" -q -- chroot /host sysctl -n vm.max_map_count 2>/dev/null
    done

    Every node prints 262144.

  3. Create the namespace and grant the SCCs.

    Create the namespace and grant the SCCs
    oc new-project datastrato
    oc adm policy add-scc-to-group anyuid system:serviceaccounts:datastrato
    oc adm policy add-scc-to-group nonroot-v2 system:serviceaccounts:datastrato

    The grants apply to every service account the chart creates in the namespace.

  4. Find your hostnames. The Router serves every hostname under the cluster's apps domain with a wildcard certificate, so no DNS record and no cert-manager are needed.

    Read the cluster apps domain
    oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}{"\n"}'

    Use gravitino. followed by that domain as UI_HOST in install step 1.

Changes to the install steps on OpenShift.

  • Step 1: set INGRESS_CLASS=openshift-default. TLS_SECRET is not used.

  • Step 4: the chart's registry-token renewal cannot run on OpenShift in this release, so create the pull Secret yourself instead of ecr-refresh-iam:

    Create the pull Secret by hand
    oc -n "$NAMESPACE" create secret docker-registry image-pull-secret \
      --docker-server="$REGISTRY" --docker-username=AWS \
      --docker-password="$(AWS_ACCESS_KEY_ID="$AKID" AWS_SECRET_ACCESS_KEY="$SKEY" \
        aws ecr get-login-password --region ap-northeast-1)"
    unset AKID SKEY

    The token in that Secret expires after about twelve hours, after which new pulls fail. For anything beyond an evaluation, mirror the images into your own registry; see Air-Gapped Installation.

  • Step 5: in my-values.yaml, replace the global block with the one below, and in the ingress block replace tls: and its entries with tls: [] and add the Router annotation:

    my-values.yaml
    global:
      imagePullSecrets:
        - name: image-pull-secret
      pullSecrets:
        create: false
        name: image-pull-secret
        ecrRefresh:
          enabled: false
    my-values.yaml
    ingress:
      enabled: true
      className: "openshift-default"
      annotations:
        route.openshift.io/termination: edge
      # hosts: unchanged
      tls: []

    The Router turns each Ingress into an edge-terminated Route on the cluster certificate. Without the global.imagePullSecrets entry, the render in step 6 fails on the OpenSearch pull Secret check.

  • Step 9: there is no Certificate resource to check. oc get routes -n "$NAMESPACE" lists the Routes, each with edge termination.

  • Uninstall: run helm uninstall with --no-hooks. The chart's cleanup Job is not admitted on OpenShift.

Issue certificates with cert-manager​

Skip this if you already have a TLS Secret for your hostname. Otherwise, install cert-manager and a Let's Encrypt issuer. The hostname must resolve to your ingress controller and port 80 must be reachable from the internet, since Let's Encrypt validates the hostname over it.

Install cert-manager and a Let's Encrypt issuer
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager -n cert-manager --create-namespace \
  --version v1.16.2 --set crds.enabled=true
kubectl -n cert-manager wait --for=condition=Available deploy --all --timeout=180s
cat > letsencrypt-issuer.yaml << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
EOF
kubectl apply -f letsencrypt-issuer.yaml
sleep 15
kubectl get clusterissuer letsencrypt

The issuer shows True in the READY column. If your ingress class is not traefik, change ingressClassName to match. In install step 5, add cert-manager.io/cluster-issuer: letsencrypt under ingress.annotations, and choose any name for TLS_SECRET in step 1; cert-manager creates that Secret.

During issuance, cert-manager runs one short-lived cm-acme-http-solver pod per hostname. They end in Error when cert-manager stops them after validation, which is expected.

Install​

  1. Set your variables. Every later command reads these, so this is the only place you type your own values. Keep this shell open for the whole procedure. If you open a new shell, run this block again first.

    Set the install variables
    VERSION={chart_version}
    REGISTRY=730335553010.dkr.ecr.ap-northeast-1.amazonaws.com
    NAMESPACE=datastrato
    RELEASE=enterprise
    PKG={path_to_release_package}
    UI_HOST={ui_hostname}
    INGRESS_CLASS={ingress_class}
    TLS_SECRET={tls_secret_name}
    METALAKE={metalake_name}
    ADMIN_PASSWORD='{admin_password}'

    Choose an admin password without quotes, backslashes or $, since it is written into a YAML file in step 5.

    METALAKE is the name of the first metalake you create in step 10. The chart needs it at install time because the Iceberg REST service and the MCP server each serve one metalake.

  2. Read the pull key. The CSV holds an access key ID and a secret access key, and the command below reads both into the shell without printing them.

    Read the pull key into the shell
    read -r AKID SKEY < <(awk -F',' 'NR==1{for(i=1;i<=NF;i++){h=tolower($i); gsub(/\r/,"",h); if(h ~ /access key id/)a=i; if(h ~ /secret access key/)s=i}} NR==2{gsub(/\r/,""); print $a, $s}' "$PKG/gravitino-ecr-pull-user_accessKeys.csv")
    echo "key id: ${AKID:0:4}...  secret length: ${#SKEY}"

    The secret length should print 40. If either value is empty, check the PKG path.

  3. Log in to the registry and pull the chart. The registry username is always AWS. The password is a token exchanged from the pull key, valid for about twelve hours. When it expires, run the login again; the key itself does not expire.

    Log in to the registry and pull the chart
    AWS_ACCESS_KEY_ID="$AKID" AWS_SECRET_ACCESS_KEY="$SKEY" \
      aws ecr get-login-password --region ap-northeast-1 \
      | helm registry login --username AWS --password-stdin "$REGISTRY"

    helm pull "oci://$REGISTRY/datastratocharts/enterprise-helm" --version "$VERSION"
    helm show values "./enterprise-helm-$VERSION.tgz" > "chart-defaults-$VERSION.yaml"
    ls -l "enterprise-helm-$VERSION.tgz"

    chart-defaults-$VERSION.yaml is the chart's full set of defaults, kept for reference. Do not edit it. Your settings go in a separate file in step 5, which holds only what you change, so a new chart version never forces you to merge.

  4. Create the namespace and the pull Secret. The cluster pulls images on its own schedule, on node restarts and rescheduling as well as at install, so it cannot rely on the token from step 3. The chart instead keeps the pull key in a Secret and renews the registry token itself, before the install and every six hours after.

    Create the namespace and pull Secret
    kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
    kubectl -n "$NAMESPACE" create secret generic ecr-refresh-iam \
      --from-literal=aws-access-key-id="$AKID" \
      --from-literal=aws-secret-access-key="$SKEY" \
      --dry-run=client -o yaml | kubectl apply -f -
    unset AKID SKEY
    kubectl -n "$NAMESPACE" get secret ecr-refresh-iam

  5. Write your values file. Run the block as-is. It substitutes your variables from step 1 and generates the session secret the UI signs cookies with.

    my-values.yaml
    cat > my-values.yaml << EOF
    # my-values.yaml
    global:
      pullSecrets:
        create: false
        name: image-pull-secret
        ecrRefresh:
          enabled: true
          existingSecret: ecr-refresh-iam

    gravitino:
      authorization:
        enable: true
        serviceAdmins: admin
      authenticators: "basic"
      admin:
        username: admin
        password: "$ADMIN_PASSWORD"
      icebergRest:
        dynamicConfigProvider:
          metalake: "$METALAKE"
      env:
        - name: GRAVITINO_MEM
          value: "-Xms4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:+ExitOnOutOfMemoryError"

    mcp:
      enabled: true
      metalake: "$METALAKE"

    enterpriseWeb:
      env:
        - name: NEXTAUTH_URL
          value: "https://$UI_HOST"
        - name: NEXTAUTH_SECRET
          value: "$(openssl rand -hex 32)"
        - name: AUTH_TRUST_HOST
          value: "true"

    ingress:
      enabled: true
      className: "$INGRESS_CLASS"
      hosts:
        - host: $UI_HOST
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: '{{ include "enterprise.fullname" . }}-ui'
                  port:
                    number: 3000
            - path: /mcp
              pathType: Prefix
              backend:
                service:
                  name: '{{ include "enterprise.fullname" . }}-mcp'
                  port:
                    number: 8000
      tls:
        - secretName: $TLS_SECRET
          hosts:
            - $UI_HOST
    EOF
    grep -n -E 'metalake|host:|secretName|password' my-values.yaml

    The grep shows your hostname, metalake, TLS Secret and password in place. If any line shows an empty value, a variable from step 1 was not set.

    Run this step once. Running it again overwrites later edits and generates a new session secret, which signs everyone out of the UI. From here on, edit my-values.yaml directly.

    What each block does:

    SettingPurpose
    global.pullSecretsUses the Secret from step 4 and renews the registry token automatically
    gravitino.authenticators, gravitino.adminLocal password sign-in with an admin account; add your identity provider later, see Authentication
    gravitino.icebergRest.dynamicConfigProviderThe metalake the Iceberg REST service serves; the install is refused without it
    gravitino.env GRAVITINO_MEMServer heap and metaspace; the chart default metaspace is too small for sustained use
    mcp.metalakeThe metalake the MCP server serves
    enterpriseWeb.envThe address the UI is reached at and its session signing secret
    ingressPublishes the UI and the MCP server on your hostname

    If you use cert-manager rather than a pre-made TLS Secret, add cert-manager.io/cluster-issuer: {issuer_name} under ingress.annotations and cert-manager creates the Secret named in tls.

  6. Check the render. Rendering validates your values against the cluster without installing anything. Every configuration error the chart knows how to detect is reported here.

    Render the chart against the cluster
    helm template "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$PKG"/gravitino-license-prod-*.txt)" \
      --dry-run=server > /tmp/render.yaml && echo "RENDER OK"

    It must print RENDER OK. Otherwise the message names the value that is wrong or missing; fix my-values.yaml and run it again.

    The license key is passed on the command line rather than stored in the values file, and tr strips the file's trailing newline, which the license check otherwise rejects as malformed.

  7. Install.

    Install the chart
    helm upgrade --install "$RELEASE" "./enterprise-helm-$VERSION.tgz" -n "$NAMESPACE" -f my-values.yaml \
      --set-string gravitino.license.key="$(tr -d '\n' < "$PKG"/gravitino-license-prod-*.txt)" \
      --timeout 15m 2>&1 | tee helm-install.log

    Helm prints the release notes when it finishes. The pods are still starting at that point.

  8. Wait for the pods.

    Wait for the Gravitino rollout
    kubectl -n "$NAMESPACE" rollout status deployment/"$RELEASE"-gravitino --timeout=15m
    kubectl -n "$NAMESPACE" get pods

    When the rollout reports success, every pod should be Running with all containers ready, except the registry-token Job and the schema Job, which show Completed. Expect these workloads: the Gravitino server, its PostgreSQL, the UI, the MCP server, the metrics service, OpenSearch, Marquez, and Marquez web. The metrics service and the UI sit at 0/1 for about a minute after starting, which is normal.

    Anything in ImagePullBackOff, CrashLoopBackOff or Init:Error has a cause listed under Troubleshoot.

  9. Check the certificate and the server.

    Check the certificate and the server
    kubectl -n "$NAMESPACE" get certificate 2>/dev/null
    kubectl -n "$NAMESPACE" port-forward svc/"$RELEASE"-gravitino 8090:8090 >/dev/null 2>&1 &
    sleep 3
    curl -s -u "admin:$ADMIN_PASSWORD" http://localhost:8090/api/metalakes | jq .

    If you use cert-manager, the certificate shows True in the READY column. The curl returns {"code": 0, "metalakes": []}. An empty list is correct, since the install creates no metalakes.

  10. Create the metalake. Use the same name you set as METALAKE in step 1.

    Create the metalake
    curl -s -u "admin:$ADMIN_PASSWORD" -X POST -H "Content-Type: application/json" \
      -d "{\"name\":\"$METALAKE\",\"comment\":\"\",\"properties\":{}}" \
      http://localhost:8090/api/metalakes | jq '{code, name: .metalake.name}'

    The response shows "code": 0 and your metalake name.

  11. Sign in to the UI. Open https://{ui_hostname} in a browser and sign in as admin with the password from step 1.

The install is complete. Keep the port forward running if you are continuing to Getting started to connect a first catalog; otherwise stop it with kill %1.

Prepare for production​

The procedure above is the same one a production install follows. Production is not a separate install; it is the same my-values.yaml with the changes below, applied by running step 7 again. An evaluation can therefore grow into a production deployment without being reinstalled, apart from moving the entity store, which is easiest to decide before you load real metadata.

AreaEvaluation, as installed aboveProduction
LicenseTrial key with a short expiryProduction key from your account team; replace it on the command line in step 7
Sign-inLocal admin account onlyYour identity provider, with oauth,basic so the local admin account remains for recovery; see Configure
Users and groupsCreated by handProvisioned from your directory through SCIM; see Configure
Admin passwordWritten into my-values.yamlHeld in a Secret you create, named in gravitino.admin.existingSecret, so the values file carries no credential
Entity storePostgreSQL deployed by the chartA database you operate and back up, with gravitino.postgresql.enabled: false; see Relational backend storage
ImagesPulled from the Datastrato release registryMirrored into your own registry with the offline toolkit, with global.registry pointed at it; see Air-Gapped Installation
MemoryScheduled without reservationsgravitino.resources with requests.memory equal to limits.memory, so the server is the last pod evicted under node pressure
BackupsNoneThe entity store database. The search index is rebuilt from the entity store and needs no backup
ReplicasOneOne. Multiple replicas are not supported in this release

The TLS requirement is the same for both: a certificate the browser trusts. An evaluation on a self-signed certificate fails at sign-in, so it cannot be deferred to production.

Configure​

Every change is made in my-values.yaml and applied by running step 7 again with the same command. Keep the license on the command line each time.

To do thisSet
Sign in through your identity providergravitino.authenticators: "oauth,basic", gravitino.authenticator.oauth.*, and gravitino.browserSso; see Authentication
Publish the Iceberg REST service to enginesingress.iceberg.enabled, ingress.iceberg.host, and ingress.iceberg.tls
Set the Iceberg REST default cataloggravitino.icebergRest.dynamicConfigProvider.defaultCatalogName
Provision users and groups from a directorygravitino.scim.enabled and gravitino.scim.ingress; see SCIM provisioning
Preview table data through your own Trinogravitino.preview.enabled and gravitino.preview.jdbcUrl
Supply JDBC drivers that cannot ship in the image, such as MySQLgravitino.jdbcDrivers.enabled and gravitino.jdbcDrivers.mode
Use your own database for the entity storeSee Relational backend storage

The chart deploys PostgreSQL for the entity store by default. Its data sits on a persistent volume that survives upgrades and uninstalls. For production, a database you operate and back up is the better choice.

The admin password is set once, at first install. The chart keeps it in a Secret that survives upgrades, so changing gravitino.admin.password later has no effect; change the password through the UI or the API instead.

gravitino.env replaces the chart's environment list rather than adding to it. When you add a variable, keep the GRAVITINO_MEM entry in the same list.

Google Cloud credentials​

Catalogs on Google Cloud Storage or BigQuery read a service account key from the pod. Create the Secret before installing or upgrading, since the chart refuses to render a mount for a Secret that does not exist.

Create the service account key Secret
kubectl -n "$NAMESPACE" create secret generic gcs-service-account \
  --from-file=key.json={path_to_service_account_key}
my-values.yaml
gravitino:
  env:
    - name: GRAVITINO_MEM
      value: "-Xms4g -Xmx4g -XX:MaxMetaspaceSize=2g -XX:+ExitOnOutOfMemoryError"
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /etc/gcs/key.json
  extraVolumes:
    - name: gcs-credentials
      secret:
        secretName: gcs-service-account
        items:
          - key: key.json
            path: key.json
  extraVolumeMounts:
    - name: gcs-credentials
      mountPath: /etc/gcs
      readOnly: true

S3 and Azure credentials are catalog properties rather than pod configuration. See the individual catalog pages under Connect sources.

Air-gapped installation​

The offline toolkit in the release package copies every image the chart uses into your own registry. Its README covers the download, upload and push modes. Push the chart to the same registry, then set global.registry in my-values.yaml to redirect every image there. With images in your own registry, global.pullSecrets.ecrRefresh and the ecr-refresh-iam Secret are no longer needed; supply whatever pull credential your own registry requires instead.

Upgrade​

Pull the new chart version (steps 1 to 3 with the new VERSION), read its chart-release-summary-{version}.md for new required values, then run steps 6 and 7 with your existing my-values.yaml. Entity store schema changes run automatically in a Job during the upgrade. See Upgrades.

Uninstall​

Uninstall the release
helm uninstall "$RELEASE" -n "$NAMESPACE"

Uninstalling leaves the persistent volumes of the built-in PostgreSQL and OpenSearch in place, along with the admin credentials Secret, so a reinstall under the same release name picks up the existing data. For a clean slate, delete the namespace as well.

Delete the namespace
kubectl delete namespace "$NAMESPACE"

Nothing in your connected sources is affected either way, since Gravitino never held that data.

Removing an evaluation cluster. On EKS, AKS and GKE, delete the namespace and wait for its volumes to go before you delete the cluster. Deleting the cluster first can leave the PostgreSQL and OpenSearch cloud disks behind, still billed, and on EKS can also leave the ingress load balancer. Delete the ingress controller release too, so its load balancer is removed while the cluster can still clean it up.

Remove the namespace and ingress controller
kubectl delete namespace "$NAMESPACE" --timeout=5m
helm uninstall traefik -n traefik
kubectl get pv

kubectl get pv prints No resources found once every volume is deleted. Then delete the cluster with your platform's command (eksctl delete cluster, az group delete, gcloud container clusters delete, or rosa delete cluster). On ROSA, also delete the operator roles, the OIDC config and the network stack afterwards, since the network's NAT gateway is billed by the hour.

Troubleshoot​

SymptomCause and fix
Render fails saying a license key is requiredThe --set-string gravitino.license.key argument is missing or the file path matched nothing. Check ls "$PKG"/gravitino-license-prod-*.txt
Render fails naming dynamicConfigProvider.metalakegravitino.icebergRest.dynamicConfigProvider.metalake is unset in my-values.yaml
Render fails naming a Secret and giving a kubectl create secret commandA Secret referenced by extraVolumes does not exist yet. Run the command it prints
Server logs report INVALID_FORMAT for the licenseThe key was passed with its trailing newline. Pass it through tr -d '\n' as in step 7
Pods in ImagePullBackOffThe ecr-refresh-iam Secret is missing or holds the wrong key. Repeat step 4, then delete the failing pods
OpenSearch restartingvm.max_map_count is below 262144 on its node
Sign-in returns to the login page, or shows a configuration errorThe browser does not trust the certificate, or NEXTAUTH_URL does not match the address in the browser
Gravitino pod restarts after running for a day or moreMetaspace exhausted. Keep -XX:MaxMetaspaceSize=2g in GRAVITINO_MEM
curl in step 9 returns connection refusedThe port forward has stopped. Run the port-forward line again
PersistentVolumeClaims stay Pending on EKSThe EBS CSI driver add-on is missing, or no StorageClass is marked default. See Amazon EKS steps 1 and 2
A pod stays Pending with a volume node affinity conflict on EKSNo node is running in the Availability Zone of its EBS volume. Add a node group in that zone
Certificate never becomes readyLet's Encrypt cannot reach port 80 on the hostname. Check that DNS points at the ingress controller and port 80 is open
az aks create fails with a quota errorThe subscription's vCPU quota is too low for the node size and count. See Azure AKS step 1
OpenSearch never starts on GKE and the max-map-count DaemonSet is rejectedThe cluster is GKE Autopilot, which rejects privileged pods. Use a GKE Standard cluster
On OpenShift, a Job or pod is never created and events show unable to validate against any security context constraintThe namespace lacks the SCC grants. See Red Hat OpenShift step 3
On OpenShift, the registry-token Job's ecr-get-login-password container shows container has runAsNonRoot and image will run as rootRegistry-token renewal is enabled. Disable it and supply the pull Secret as described under Red Hat OpenShift
Render fails with opensearch.imagePullSecrets lists "image-pull-secret", but this release will not create that SecretRegistry-token renewal is disabled and the Secret is supplied by hand. Add image-pull-secret under global.imagePullSecrets
kubectl or helm fails with tls: bad record MACPackets are corrupted on the network path from your machine, usually an MTU mismatch behind a VPN or under WSL. Lower the interface MTU, for example sudo ip link set dev eth0 mtu 1350, and retry
Last updated on