BigQuery catalog

The BigQuery catalog connects Gravitino to a Google BigQuery project through JDBC, with the provider name jdbc-bigquery. BigQuery datasets appear as schemas and BigQuery tables as tables, and Gravitino reads and writes through to BigQuery, so the project stays authoritative.

Gravitino authenticates as a Google Cloud service account, using that account's JSON key file.

Quick start

The example connects a Google Cloud project as the catalog bigquery_catalog and lists its datasets as schemas. Two things must be on the Gravitino server first:

Supply the Simba driver. The Simba BigQuery JDBC driver does not ship in the image. Enable it with gravitino.jdbcDrivers.bigquery in my-values.yaml, as described in JDBC driver.
my-values.yaml
```
gravitino:
  jdbcDrivers:
    enabled: true
    bigquery:
      enabled: true
```
Mount the service account key. The catalog takes the path to the key file, not its contents, so the file must exist in the server pod. Mount it from a Secret as described in Google Cloud credentials, which places it at /etc/gcs/key.json.

UI
REST

Start the catalog. Open Connect → Catalogs and click + Catalog. Enter bigquery_catalog as the Catalog name, keep Relational as the Catalog type, and choose Google BigQuery under Databases and warehouses in Select provider.
Enter the connection. Under Connection, enter {project_id} as the Project ID, jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443 as the URL, the service account email as the User, and /etc/gcs/key.json as the Password. The UI requires a URL, and the connector adds the project and credentials to it. Advanced holds a Properties list for the proxy properties. The UI fills in jdbc-driver from the provider.
Test and create it. Click Test connection, then Create catalog.
List the datasets. In Explore, open bigquery_catalog. Each BigQuery dataset in the project comes back as a schema, across every region the project uses.

Create the catalog. Name the project, the service account, and the key file path. The connector defaults jdbc-url and jdbc-driver, so the request can omit both.

Create a BigQuery catalog
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "bigquery_catalog",
    "type": "RELATIONAL",
    "provider": "jdbc-bigquery",
    "comment": "BigQuery catalog",
    "properties": {
      "project-id": "{project_id}",
      "jdbc-user": "{service_account_email}",
      "jdbc-password": "/etc/gcs/key.json"
    }
  }'

List the datasets. Each BigQuery dataset in the project comes back as a schema, across every region the project uses.

List the schemas in the catalog
curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/bigquery_catalog/schemas" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Authorization: Bearer $TOKEN"

Catalog properties

project-id and the proxy properties are BigQuery's own. The connector also gives jdbc-url, jdbc-driver, jdbc-user, and jdbc-password BigQuery-specific meanings and defaults, listed here. The pool properties are described in Shared properties.

Property	Description	Default	Required
`project-id`	Google Cloud project ID	(none)	Yes
`jdbc-user`	Service account email	(none)	Yes
`jdbc-password`	Path to the service account's JSON key file on the Gravitino server	(none)	Yes
`jdbc-url`	JDBC URL. Built from the properties above when not set	`jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443`	No
`jdbc-driver`	JDBC driver class	`com.simba.googlebigquery.jdbc42.Driver`	No
`proxy-host`	Proxy host or IP address	(none)	No
`proxy-port`	Proxy port, from 1 to 65535. Required when `proxy-host` is set	(none)	No
`proxy-username`	Proxy username	(none)	No
`proxy-password`	Proxy password. Hidden in catalog responses	(none)	No

A jdbc-url that already carries ProjectId= and either OAuthServiceAcctEmail= or OAuthType= is used as written, and jdbc-user and jdbc-password are then not required. Use that form only when the Simba driver needs connection options the properties above do not cover.

Schemas

A schema is a BigQuery dataset. Dropping a schema that still holds tables fails unless the drop is a cascade, which removes the tables with it. INFORMATION_SCHEMA and SYS are reserved and cannot be used as schema names.

Property Name	Description	Changeable
`location`	Dataset location, such as `us`, `eu`, or `asia-northeast1`	No
`default_collation`	Default collation for `STRING` columns, such as `und:ci` for case-insensitive	No
`description`	Dataset description	Yes
`friendly_name`	Display name	Yes
`labels`	Labels as a JSON array, `[{"key":"value"}]`	Yes
`tags`	IAM tags as a JSON array, `[{"key":"value"}]`	Yes
`default_table_expiration_days`	Days after which new tables expire	Yes
`default_partition_expiration_days`	Days after which new partitions expire	Yes
`is_case_insensitive`	Whether table and column names in the dataset are case-insensitive	Yes
`storage_billing_model`	`LOGICAL` or `PHYSICAL`	Yes
`max_time_travel_hours`	Time travel window, a multiple of 24 from 48 to 168	Yes
`default_kms_key_name`	Default Cloud KMS key for tables in the dataset	Yes
`default_rounding_mode`	`ROUND_HALF_EVEN` or `ROUND_HALF_AWAY_FROM_ZERO`	Yes
`failover_reservation`	Reservation to use in a failover	Yes
`is_primary`	Whether this dataset is the primary replica	Yes
`primary_replica`	Name of the replica to make primary	Yes

Tables

Properties

Property Name	Description
`description`	Table description
`friendly_name`	Display name
`labels`	Labels as a JSON array, `[{"key":"value"}]`
`tags`	IAM tags as a JSON array, `[{"key":"value"}]`
`clustering_fields`	Comma-separated clustering columns, at most four
`partition_expiration_days`	Days after which partitions expire; fractional values are allowed
`require_partition_filter`	Whether queries must filter on the partition column
`expiration_timestamp`	When the table expires, in ISO 8601, such as `2027-12-31T23:59:59Z`
`kms_key_name`	Cloud KMS key for the table
`default_rounding_mode`	Rounding mode for `NUMERIC` and `BIGNUMERIC` columns
`max_staleness`	How far behind current time reads may be, as `INTERVAL "4:0:0" HOUR TO SECOND`
`enable_change_history`	Capture change history for the `CHANGES` function
`enable_fine_grained_mutations`	Enable fine-grained DML optimization
`storage_uri`	For a managed Iceberg table, the `gs://` location of its data
`file_format`	For a managed Iceberg table, `PARQUET`
`table_format`	For a managed Iceberg table, `ICEBERG`

Indexes, sort orders, and distribution

BigQuery has no indexes or constraints, so a table created with an index is rejected. Sort orders and distribution are rejected as well. Clustering takes their place and is set with the clustering_fields property.

Partitioning

A table can be partitioned on one column.

Transform	Column Types
`day`	`DATE`, `TIMESTAMP`, `DATETIME`
`month`	`DATE`, `TIMESTAMP`, `DATETIME`
`year`	`DATE`, `TIMESTAMP`, `DATETIME`
`hour`	`TIMESTAMP`, `DATETIME`
`identity`	`DATE`, `TIMESTAMP`, `DATETIME`

Identity partitioning on a TIMESTAMP or DATETIME column partitions by its date, the same as day. Partitioning on an integer column is not supported by the connector yet, although BigQuery itself supports it through integer ranges.

Supported changes

Change	Notes
Rename a table	Within the same dataset
Update the comment
Set a property
Add a column
Delete a column
Rename a column
Change a column's type	Only where BigQuery allows the conversion
Make a column nullable	A nullable column cannot be made `NOT NULL` again

Any other change is rejected.

Identifier case and quoting

Dataset, table, and column names start with a letter or underscore, contain only letters, digits, and underscores, and are 1 to 1024 characters long. __TABLES__, __TABLES_SUMMARY__, __PARTITIONS__, and __PARTITIONS_SUMMARY__ are reserved table names. Gravitino quotes names with backticks when it generates SQL. Whether names are case-sensitive is decided per dataset by is_case_insensitive.

Type mapping

Gravitino reads each BigQuery type as the Gravitino type in its row, and creates each Gravitino type as the BigQuery type in its row, except where the list under the table says otherwise.

BigQuery type	Gravitino type
`BOOL`	`boolean`
`INT64`	`long`
`FLOAT64`	`double`
`NUMERIC(p, s)`	`decimal(p, s)`, or `decimal(38, 9)` when precision is unknown
`STRING`	`string`
`BYTES`	`binary`
`DATE`	`date`
`TIME`	`time`
`DATETIME`	`timestamp`
`TIMESTAMP`	`timestamp_tz`
`BIGNUMERIC`, `GEOGRAPHY`, `JSON`, `STRUCT`, `RANGE`, and any other type	External type, carrying the BigQuery type name

When Gravitino creates a column:

byte, short and integer become INT64, and read back as long.
float becomes FLOAT64, and reads back as double.
varchar and char become STRING, and read back as string.
An external or unparsed type becomes the type as written, such as ARRAY<INT64> or STRUCT<a INT64>.

Error mapping

BigQuery Error	Gravitino Error
Table not found	`NoSuchTableException`
Dataset not found	`NoSuchSchemaException`
Table already exists	`TableAlreadyExistsException`
Dataset already exists	`SchemaAlreadyExistsException`
Permission or access denied	A runtime error naming the denial
Invalid request	A runtime error naming the problem

Quick start​

Catalog properties​

Schemas​

Tables​

Properties​

Indexes, sort orders, and distribution​

Partitioning​

Supported changes​

Identifier case and quoting​

Type mapping​

Error mapping​

Related​