Doris Catalog

Overview

Apache Gravitino provides the ability to manage Apache Doris metadata through JDBC connection.

Gravitino saves some system information in schema and table comments, like (From Gravitino, DO NOT EDIT: gravitino.v1.uid1078334182909406185), do not change or remove this message.

Catalog

Catalog Capabilities

Gravitino catalog corresponds to the Doris instance.
Supports metadata management of Doris (1.2.x, 3.0.x, 4.0.x).
Supports table index (PRIMARY_KEY, UNIQUE_KEY, INVERTED, BITMAP (legacy), ANN/VECTOR).
Supports column default value.

Catalog Properties

Pass to a Doris data source any property that isn't defined by Gravitino by adding gravitino.bypass. prefix as a catalog property. For example, catalog property gravitino.bypass.maxWaitMillis will pass maxWaitMillis to the data source property.

Check the relevant data source configuration in data source properties for more details.

Besides the common catalog properties, the Doris catalog has the following properties:

Configuration item	Description	Default value	Required
`jdbc-url`	JDBC URL for connecting to the database. For example, `jdbc:mysql://localhost:9030`	(none)	Yes
`jdbc-driver`	The driver of the JDBC connection. For example, `com.mysql.jdbc.Driver`.	(none)	Yes
`jdbc-user`	The JDBC user name.	(none)	Yes
`jdbc-password`	The JDBC password.	(none)	Yes
`jdbc.pool.min-size`	The minimum number of connections in the pool. `2` by default.	`2`	No
`jdbc.pool.max-size`	The maximum number of connections in the pool. `10` by default.	`10`	No
`jdbc.pool.max-wait-ms`	The maximum Duration that the pool will wait for a connection to be returned. `30000` by default.	`30000`	No

Before using the Doris Catalog, you must download the corresponding JDBC driver to the catalogs/jdbc-doris/libs directory. Gravitino doesn't package the JDBC driver for Doris due to licensing issues.

Driver Version Compatibility

The Doris catalog includes driver version compatibility checks for datetime precision calculation:

MySQL Connector/J versions >= 8.0.16: Full support for datetime precision calculation
MySQL Connector/J versions < 8.0.16: Limited support - datetime precision calculation returns null with a warning log

This limitation affects the following datetime types:

DATETIME(p) - datetime precision

When using an unsupported driver version, the system will:

Continue to work normally with default precision (0)
Log a warning message indicating the driver version limitation
Return null for precision calculations to avoid incorrect results

Example warning log:

WARN: MySQL driver version mysql-connector-java-8.0.11 is below 8.0.16, 
columnSize may not be accurate for precision calculation. 
Returning null for DATETIME type precision. Driver version: mysql-connector-java-8.0.11

Recommended driver versions:

mysql-connector-java-8.0.16 or higher

Catalog Operations

Refer to Manage Relational Metadata Using Gravitino for more details.

Sensitive catalog properties such as jdbc-user and jdbc-password are hidden from the load catalog response. Use the credential vending API to retrieve them at runtime.

Schema

Schema Capabilities

Gravitino's schema concept corresponds to the Doris database.
Supports creating schema.
Supports dropping schema.

Schema Properties

Support schema properties, including Doris database properties and user-defined properties.

Schema Operations

Refer to Manage Relational Metadata Using Gravitino for more details.

Table

Table Capabilities

Gravitino's table concept corresponds to the Doris table.
Supports index.
Supports column default value.

Table Column Types

Gravitino Type	Doris Type
`Boolean`	`Boolean`
`Byte`	`TinyInt`
`Short`	`SmallInt`
`Integer`	`Int`
`Long`	`BigInt`
`Float`	`Float`
`Double`	`Double`
`Decimal`	`Decimal`
`Date`	`Date`/`DateV2`
`Timestamp[(p)]`	`Datetime[(p)]`
`VarChar`	`VarChar`
`FixedChar`	`Char`
`String`	`String`
`Binary`	`Binary`/`VarBinary`
`ExternalType("json")`	`JSON`
`ExternalType("variant")`	`Variant`
`ExternalType("ipv4")`	`IPv4`
`ExternalType("ipv6")`	`IPv6`
`ExternalType("largeint")`	`LargeInt`
`ExternalType("bitmap")`	`Bitmap`
`ExternalType("hll")`	`HLL`

Doris doesn't support Gravitino Fixed Timestamp_tz IntervalDay IntervalYear Union UUID type. The data types other than those listed above are mapped to Gravitino's Unparsed Type that represents an unresolvable data type.

Doris array, map, and struct types are loaded as ExternalType with the full type string preserved (e.g. array<int(11)>). They are not resolved into Gravitino native composite types (ListType, MapType, StructType). The type identifier in ExternalType is always lowercase (e.g. "json", not "JSON"), matching Doris JDBC metadata behavior.

DateV2 type: Doris 1.2+ (required on 4.0.x where disable_datev1=true)
Binary / VarBinary type: Doris 4.0+ (not available on 3.x)
Auto-Increment column: Doris 2.1+
INVERTED index: Doris 3.0+
ANN / VECTOR index: Doris 4.0.6+

Table Column Auto-Increment

Auto-increment columns are supported on Doris 2.1+. Gravitino validates the Doris version at table creation time and rejects auto-increment columns on older versions.

Doris enforces the following constraints (violations are rejected by the Doris server):

The table must use UNIQUE KEY or DUPLICATE KEY model.
The auto-increment column must be BIGINT NOT NULL with no DEFAULT value.
Each table can have at most one auto-increment column.

Gravitino currently supports creating UNIQUE KEY tables via the UNIQUE_KEY index type. To create a DUPLICATE KEY table, omit key indexes from the table definition — Doris defaults to the DUPLICATE model when no key is specified.

JSON
Java

{
  "columns": [
    {
      "name": "id",
      "type": "long",
      "nullable": false,
      "autoIncrement": true
    }
  ],
  "indexes": [
    {
      "indexType": "unique_key",
      "name": "id_key",
      "fieldNames": [["id"]]
    }
  ]
}

Column column = Column.of("id", Types.LongType.get(), "", false, true, null);
Index[] indexes = new Index[] {
    Indexes.of(Index.IndexType.UNIQUE_KEY, "id_key", new String[][]{{"id"}}, Map.of())
};

Table Properties

Doris table properties can be set when creating a table. Only Doris built-in table properties are supported; user-defined properties are not supported.

Property Name	Description	Default Value	Required	Reserved	Immutable
`replication_num`	The number of replications for the table. If not specified and the number of backend servers less than 3, then the default value is 1; If BE ≥ 3, the server-side default (3) will be used.	`1` or `3`	No	No	No
`compression`	The compression type for the table. Supported values: `ZSTD`, `LZ4`, `LZ4F`, `ZLIB`. Deprecated as a table-level property in Doris 4.0+. Cannot be changed after table creation.	(none)	No	No	Yes
`bloom_filter_columns`	Comma-separated list of columns for which bloom filter indexes are created.	(none)	No	No	No
`storage_policy`	The name of the storage policy for cold-hot separation.	(none)	No	No	No
`light_schema_change`	Whether light schema change is enabled for the table. Can be modified via ALTER TABLE SET.	`true`	No	No	No
`enable_unique_key_merge_on_write`	Whether merge-on-write is enabled for Unique Key tables. Must be set at CREATE TABLE time; cannot be changed after creation.	`true`	No	No	Yes

Immutable properties can be set at CREATE TABLE time but cannot be changed via ALTER TABLE. Reserved properties (none currently) are read-only and cannot be set by users.

Table Indexes

The Doris catalog supports the following index types. Each index applies to a single column.

Gravitino Index Type	Doris DDL	Doris Version
`PRIMARY_KEY`	INDEX `PRIMARY` (col) (in the INDEX clause, no USING)	1.2+
`UNIQUE_KEY`	`UNIQUE KEY(col)` (in the table model section, not INDEX clause)	1.2+
`INVERTED`	`INDEX name (col) USING INVERTED`	3.0+
`BITMAP`	`INDEX name (col)` (bare, no USING clause; write-only, see note below)	1.2+
`VECTOR`	`INDEX name (col) USING ANN`	4.0.6+

PRIMARY_KEY stays in the INDEX clause as a bare index (e.g. INDEX `PRIMARY` (`id`)), with no USING clause.
UNIQUE_KEY is emitted as a table model declaration (e.g. UNIQUE KEY(`id`)), outside the INDEX clause.
BITMAP is a write-only legacy type for backward compatibility with Doris 1.2.x. The write path generates a bare INDEX (no USING clause), but the read path maps it back to INVERTED because Doris 4.0.6 removed BITMAP from the grammar. Creating a BITMAP index and reading it back will show INVERTED.

Primary Key example:

JSON
Java

{
  "indexes": [
    {
      "indexType": "primary_key",
      "name": "PRIMARY",
      "fieldNames": [["id"]]
    }
  ]
}

Index[] indexes = new Index[] {
    Indexes.of(IndexType.PRIMARY_KEY, "PRIMARY", new String[][]{{"id"}}, Map.of())
};

Inverted Index example (Doris 3.0+):

JSON
Java

{
  "indexes": [
    {
      "indexType": "inverted",
      "name": "idx_name",
      "fieldNames": [["name"]]
    }
  ]
}

Index[] indexes = new Index[] {
    Indexes.of(IndexType.INVERTED, "idx_name", new String[][]{{"name"}}, Map.of())
};

Table Partitioning

The Doris catalog supports partitioned tables. Users can create partitioned tables in the Doris catalog with specific partitioning attributes. It is also supported to pre-assign partitions when creating Doris tables. Note that although Gravitino supports several partitioning strategies, Apache Doris inherently only supports these two partitioning strategies:

RANGE
LIST

The fieldName specified in the partitioning attributes must be the name of columns defined in the table.

Table Distribution

Users can also specify the distribution strategy when creating tables in the Doris catalog. The Doris catalog supports the following distribution strategies:

HASH
RANDOM

For the RANDOM distribution strategy, Gravitino uses the EVEN to represent it. More information about the distribution strategy defined in Gravitino can be found here.

Table Operations

Refer to Manage Relational Metadata Using Gravitino for more details.

Alter Table Operations

Gravitino supports these table alteration operations:

RenameTable
UpdateComment
AddColumn
DeleteColumn
UpdateColumnType
UpdateColumnPosition
UpdateColumnComment
SetProperty

Please be aware that:

Not all table alteration operations can be processed in batches.
Schema changes, such as adding/modifying/dropping columns can be processed in batches.
Supports modifying multiple column comments at the same time.
Doesn't support modifying the column type and column comment at the same time.
The schema alteration in Doris is asynchronous. You might get an outdated schema if you execute a schema query immediately after the alteration. Pause briefly after the alteration. Gravitino will surface the schema-alteration status in the schema information in an upcoming release to solve this.

Overview​

Catalog​

Catalog Capabilities​

Catalog Properties​

Driver Version Compatibility​

Catalog Operations​

Schema​

Schema Capabilities​

Schema Properties​

Schema Operations​

Table​

Table Capabilities​

Table Column Types​

Table Column Auto-Increment​

Table Properties​

Table Indexes​

Table Partitioning​

Table Distribution​

Table Operations​

Alter Table Operations​

Overview

Catalog

Catalog Capabilities

Catalog Properties

Driver Version Compatibility

Catalog Operations

Schema

Schema Capabilities

Schema Properties

Schema Operations

Table

Table Capabilities

Table Column Types

Table Column Auto-Increment

Table Properties

Table Indexes

Table Partitioning

Table Distribution

Table Operations

Alter Table Operations