BigQuery Catalog

Overview

The BigQuery catalog connects Gravitino to a Google BigQuery project through JDBC, with the provider name jdbc-bigquery. BigQuery datasets appear as schemas and BigQuery tables as tables, and Gravitino reads and writes through to BigQuery, so the project stays authoritative.

Gravitino authenticates as a Google Cloud service account, using that account's JSON key file.

Quick Start

1. Put the service account key where the server can read it. The catalog takes the path to the key file, not its contents, so the file must exist on the Gravitino server. On Kubernetes, mount it into the server pod from a secret. The Simba BigQuery JDBC driver ships inside the connector, so there is no driver to download.

2. Create the catalog. Name the project, the service account, and the key file path.

REST

curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "bigquery_catalog",
    "type": "RELATIONAL",
    "provider": "jdbc-bigquery",
    "comment": "BigQuery catalog",
    "properties": {
      "project-id": "{project_id}",
      "jdbc-user": "{service_account_email}",
      "jdbc-password": "/etc/gravitino/bigquery/{key_file}.json"
    }
  }'

Java

Map<String, String> properties = ImmutableMap.<String, String>builder()
    .put("project-id", "{project_id}")
    .put("jdbc-user", "{service_account_email}")
    .put("jdbc-password", "/etc/gravitino/bigquery/{key_file}.json")
    .build();

Catalog catalog = client.createCatalog(
    "bigquery_catalog", Catalog.Type.RELATIONAL, "jdbc-bigquery", "BigQuery catalog", properties);

3. List the datasets. Each BigQuery dataset in the project comes back as a schema, across every region the project uses.

Catalog Properties

Property Name	Description	Default Value	Required
project-id	Google Cloud project ID	(none)	Yes
jdbc-user	Service account email	(none)	Yes
jdbc-password	Path to the service account's JSON key file on the Gravitino server	(none)	Yes
jdbc-url	JDBC URL. Built from the properties above when not set	jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443	No
jdbc-driver	JDBC driver class	com.simba.googlebigquery.jdbc42.Driver	No
proxy-host	Proxy host or IP address	(none)	No
proxy-port	Proxy port, from 1 to 65535. Required when proxy-host is set	(none)	No
proxy-username	Proxy username	(none)	No
proxy-password	Proxy password. Hidden in catalog responses	(none)	No

A jdbc-url that already carries ProjectId= and either OAuthServiceAcctEmail= or OAuthType= is used as written, and jdbc-user and jdbc-password are then not required. Use that form only when the Simba driver needs connection options the properties above do not cover.

The connection pool settings every JDBC catalog accepts apply here too. See Connecting a Catalog.

Schemas

A schema is a BigQuery dataset. Dropping a schema that still holds tables fails unless the drop is a cascade, which removes the tables with it. INFORMATION_SCHEMA and SYS are reserved and cannot be used as schema names.

Property Name	Description	Changeable
location	Dataset location, such as us, eu, or asia-northeast1	No
default_collation	Default collation for STRING columns, such as und:ci for case-insensitive	No
description	Dataset description	Yes
friendly_name	Display name	Yes
labels	Labels as a JSON array, [{"key":"value"}]	Yes
tags	IAM tags as a JSON array, [{"key":"value"}]	Yes
default_table_expiration_days	Days after which new tables expire	Yes
default_partition_expiration_days	Days after which new partitions expire	Yes
is_case_insensitive	Whether table and column names in the dataset are case-insensitive	Yes
storage_billing_model	LOGICAL or PHYSICAL	Yes
max_time_travel_hours	Time travel window, a multiple of 24 from 48 to 168	Yes
default_kms_key_name	Default Cloud KMS key for tables in the dataset	Yes
default_rounding_mode	ROUND_HALF_EVEN or ROUND_HALF_AWAY_FROM_ZERO	Yes
failover_reservation	Reservation to use in a failover	Yes
is_primary	Whether this dataset is the primary replica	Yes
primary_replica	Name of the replica to make primary	Yes

Tables

Table Properties

Property Name	Description
description	Table description
friendly_name	Display name
labels	Labels as a JSON array, [{"key":"value"}]
tags	IAM tags as a JSON array, [{"key":"value"}]
clustering_fields	Comma-separated clustering columns, at most four
partition_expiration_days	Days after which partitions expire; fractional values are allowed
require_partition_filter	Whether queries must filter on the partition column
expiration_timestamp	When the table expires, in ISO 8601, such as 2027-12-31T23:59:59Z
kms_key_name	Cloud KMS key for the table
default_rounding_mode	Rounding mode for NUMERIC and BIGNUMERIC columns
max_staleness	How far behind current time reads may be, as INTERVAL "4:0:0" HOUR TO SECOND
enable_change_history	Capture change history for the CHANGES function
enable_fine_grained_mutations	Enable fine-grained DML optimization
storage_uri	For a managed Iceberg table, the gs:// location of its data
file_format	For a managed Iceberg table, PARQUET
table_format	For a managed Iceberg table, ICEBERG

Indexes, Sort Orders, and Distribution

BigQuery has no indexes or constraints, so a table created with an index is rejected. Sort orders and distribution are rejected as well. Clustering takes their place and is set with the clustering_fields property.

Partitioning

A table can be partitioned on one column.

Transform	Column Types
day	DATE, TIMESTAMP, DATETIME
month	DATE, TIMESTAMP, DATETIME
year	DATE, TIMESTAMP, DATETIME
hour	TIMESTAMP, DATETIME
identity	DATE, TIMESTAMP, DATETIME

Identity partitioning on a TIMESTAMP or DATETIME column partitions by its date, the same as day. Partitioning on an integer column is not supported by the connector yet, although BigQuery itself supports it through integer ranges.

Supported Table Changes

Change	Notes
Rename a table	Within the same dataset
Update the comment
Set a property
Add a column
Delete a column
Rename a column
Change a column's type	Only where BigQuery allows the conversion
Make a column nullable	A nullable column cannot be made NOT NULL again

Any other change is rejected.

Identifier Case and Quoting

Dataset, table, and column names start with a letter or underscore, contain only letters, digits, and underscores, and are 1 to 1024 characters long. __TABLES__, __TABLES_SUMMARY__, __PARTITIONS__, and __PARTITIONS_SUMMARY__ are reserved table names. Gravitino quotes names with backticks when it generates SQL. Whether names are case-sensitive is decided per dataset by is_case_insensitive.

Type Mapping

BigQuery to Gravitino

BigQuery Type	Gravitino Type
BOOL	boolean
INT64	long
FLOAT64	double
NUMERIC(p, s)	decimal(p, s), or decimal(38, 9) when precision is unknown
STRING	string
BYTES	binary
DATE	date
TIME	time
DATETIME	timestamp
TIMESTAMP	timestamp_tz
BIGNUMERIC, GEOGRAPHY, JSON, STRUCT, RANGE, and any other type	External type, carrying the BigQuery type name

Gravitino to BigQuery

Gravitino Type	BigQuery Type
boolean	BOOL
byte, short, integer, long	INT64
float, double	FLOAT64
string, varchar, char	STRING
binary	BYTES
date	DATE
time	TIME
timestamp	DATETIME
timestamp_tz	TIMESTAMP
decimal(p, s)	NUMERIC(p, s)
External or unparsed type	The type as written, such as ARRAY<INT64> or STRUCT<a INT64>

Error Mapping

BigQuery Error	Gravitino Error
Table not found	NoSuchTableException
Dataset not found	NoSuchSchemaException
Table already exists	TableAlreadyExistsException
Dataset already exists	SchemaAlreadyExistsException
Permission or access denied	A runtime error naming the denial
Invalid request	A runtime error naming the problem