Skip to main content

Spark authentication

The Spark connector authenticates to the Gravitino server with the setting spark.sql.gravitino.authType. The Datastrato Enterprise server accepts Basic and OAuth, so those are the two modes covered here. The connector's default, simple, sends a user name with no password, which the server does not accept.

Quick start​

The example signs the connector in to a server that uses OAuth, with a client registered at the identity provider the server trusts. The connector requests a token with the client credentials grant, sending it to serverUri joined with tokenPath.

  1. Add the OAuth settings to the Spark configuration, next to the connector settings from the Spark connector Quick start.

    spark-defaults.conf
    spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
    spark.sql.gravitino.uri=https://{gravitino_host}
    spark.sql.gravitino.metalake={metalake}
    spark.sql.gravitino.authType=oauth2
    spark.sql.gravitino.oauth2.serverUri=https://{idp_host}
    spark.sql.gravitino.oauth2.tokenPath={token_path}
    spark.sql.gravitino.oauth2.credential={client_id}:{client_secret}
    spark.sql.gravitino.oauth2.scope={scope}

  2. Start Spark SQL and use a catalog. A failed sign-in stops the session at startup.

    Confirm the connector is signed in
    USE {catalog};
    SHOW DATABASES;

Basic mode​

Basic mode sends a user name and password from the server's local user store. The server must have Basic authentication enabled; see How to authenticate.

PropertyTypeDefaultDescriptionRequired
spark.sql.gravitino.authTypestringsimplebasicYes
spark.sql.gravitino.basic.usernamestring(none)User name in the local user storeYes, for Basic
spark.sql.gravitino.basic.passwordstring(none)Password for that userYes, for Basic
spark-defaults.conf
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.gravitino.uri=https://{gravitino_host}
spark.sql.gravitino.metalake={metalake}
spark.sql.gravitino.authType=basic
spark.sql.gravitino.basic.username={username}
spark.sql.gravitino.basic.password={password}

OAuth mode​

OAuth mode requests an access token from the identity provider with the client credentials grant. The Quick start shows a complete configuration.

PropertyTypeDefaultDescriptionRequired
spark.sql.gravitino.authTypestringsimpleoauth2Yes
spark.sql.gravitino.oauth2.serverUristring(none)Base address of the identity providerYes, for OAuth
spark.sql.gravitino.oauth2.tokenPathstring(none)Token endpoint path, joined to serverUriYes, for OAuth
spark.sql.gravitino.oauth2.credentialstring(none)Client credentials as {client_id}:{client_secret}Yes, for OAuth
spark.sql.gravitino.oauth2.scopestring(none)Scope to request with the tokenYes, for OAuth
Last updated on