Doris catalog

The Doris catalog connects Gravitino to an Apache Doris cluster over the MySQL protocol, exposing Doris databases as schemas and Doris tables as tables. Schemas and tables are both read-write. One catalog reaches every database on the frontend named in jdbc-url.

Gravitino records its own identifier in the comment of each schema and table it creates, as text of the form (From Gravitino, DO NOT EDIT: gravitino.v1.uid{id}), and uses it to recognize the object on later loads. Leave that text in place when you edit a comment in Doris directly.

Quick start

The example connects a Doris frontend as the catalog doris_sales and lists its databases as schemas. The catalog uses MySQL Connector/J, which does not ship in the image. Enable it with gravitino.jdbcDrivers.mysql before you start, as described in JDBC driver.

my-values.yaml
gravitino:
  jdbcDrivers:
    enabled: true
    mysql:
      enabled: true

UI
REST

Start the catalog. Open Connect → Catalogs and click + Catalog. Enter doris_sales as the Catalog name, keep Relational as the Catalog type, and choose Apache Doris under Databases and warehouses in Select provider.
Enter the connection. Under Connection, enter jdbc:mysql://{doris_fe_host}:9030 as the URL, and the User and Password Gravitino connects as. Advanced holds Cloud / region and a Properties list for any other catalog property. The UI fills in jdbc-driver itself.
Test and create it. Click Test connection, then Create catalog.
List the schemas. In Explore, open doris_sales. The schemas listed are the Doris databases, read from the frontend.

Create the catalog. The public REST endpoint does not fill in jdbc-driver, so the request names it.

Create a Doris catalog
curl -sS -X POST "https://{gravitino_host}/api/metalakes/{metalake}/catalogs" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "doris_sales",
    "type": "RELATIONAL",
    "provider": "jdbc-doris",
    "comment": "Doris sales cluster",
    "properties": {
      "jdbc-url": "jdbc:mysql://{doris_fe_host}:9030",
      "jdbc-driver": "com.mysql.cj.jdbc.Driver",
      "jdbc-user": "{user}",
      "jdbc-password": "{password}"
    }
  }'

List the schemas. They are the Doris databases, read from the frontend.

List the schemas in the catalog
curl -sS "https://{gravitino_host}/api/metalakes/{metalake}/catalogs/doris_sales/schemas" \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Authorization: Bearer $TOKEN"

Catalog properties

The Doris catalog defines no catalog properties of its own; it takes the connection, pool, and driver properties every JDBC catalog shares, listed in Shared properties.

jdbc-url points at a frontend's MySQL protocol port, and jdbc-driver is com.mysql.cj.jdbc.Driver.

Driver version compatibility

The catalog checks the Connector/J version it finds. The chart's default Connector/J, 8.0.33, meets both rules below.

Connector/J version	Effect
Below `8.0`	Catalog creation fails, naming the driver version found
`8.0.0` to `8.0.15`	The precision of a plain `DATETIME` column is unknown; it loads without precision and a warning is logged
`8.0.16` and later	Full support

The 8.0.16 boundary matters only for columns Doris reports as plain DATETIME. A column reported as DATETIME(p) takes its precision from the type string with any driver version.

Schemas

A schema in this catalog is a Doris database. Create, list, load, and drop are supported; alter is not. Listing filters out information_schema.

Doris has no schema comment syntax, so Gravitino writes the comment as a database property named comment and reads it back from SHOW CREATE DATABASE. Any other schema property is written to the PROPERTIES clause of CREATE DATABASE and read back the same way.

Dropping a schema that still holds tables fails unless the drop cascades.

warning

A cascading drop issues DROP DATABASE ... FORCE. Doris does not keep a database dropped with FORCE in its recycle bin, so the database and every table in it cannot be recovered with RECOVER.

Tables

Table create, load, list, rename, alter, and drop are supported. Purge is not, and a purge request is rejected rather than being treated as a drop. A drop issues DROP TABLE without FORCE.

Every Doris table needs a distribution, so a create without one is rejected. See Partitioning and distribution.

After creating a table, Gravitino reads its comment back from information_schema and rewrites it with ALTER TABLE ... MODIFY COMMENT if Doris did not store it, since some Doris planners discard a table comment at create time. The connecting user needs ALTER privilege only when that repair runs.

Doris applies column changes as an asynchronous schema change job. A load issued immediately after an alter can return the schema from before the change.

Properties

Two table properties are declared by the catalog. Any other property is passed to the PROPERTIES clause of CREATE TABLE, and Doris decides whether it accepts it. Properties are read back from SHOW CREATE TABLE.

Property	Description	Default
`replication_num`	The number of replicas for the table	`1` when fewer than three backends are alive, otherwise the Doris default
`replication_allocation`	The replica allocation policy for the table	None

The two properties cannot be set together. When neither is set, Gravitino runs SHOW BACKENDS to count the live backends, so the connecting user needs permission to run it; a create fails if the count cannot be read.

A set property is issued as ALTER TABLE ... SET (...), and several set properties in one request share that clause. Removing a property is rejected.

Indexes

Each index other than a unique key covers exactly one top-level column.

Gravitino index type	Doris DDL	Read back as
`PRIMARY_KEY`	INDEX `{name}` (`{column}`), no `USING`	`PRIMARY_KEY` when named `PRIMARY`
`UNIQUE_KEY`	`UNIQUE KEY({columns})`, the table model key	`UNIQUE_KEY`
`INVERTED`	INDEX `{name}` (`{column}`) USING INVERTED	`INVERTED`
`BITMAP`	INDEX `{name}` (`{column}`), no `USING`	`INVERTED`
`VECTOR`	INDEX `{name}` (`{column}`) USING ANN	`VECTOR`

A table takes at most one UNIQUE_KEY. It is emitted as the table model key rather than in the index clause, and Doris requires its columns to lead the column list. A table with no unique key is created with the Doris duplicate key model.

BITMAP is written without a USING clause for older Doris releases and is read back as INVERTED, so a bitmap index does not round-trip under its own type. Doris bloom filter indexes are read back as DATA_SKIPPING_BLOOM_FILTER but cannot be created through Gravitino, and an index type Gravitino does not recognize is read back as INVERTED.

An alter can add INVERTED, BITMAP, and VECTOR indexes and can delete any index. Primary and unique keys cannot be added after creation.

Define a unique key and an inverted index
{
  "indexes": [
    {"indexType": "UNIQUE_KEY", "name": "uk_id", "fieldNames": [["id"]]},
    {"indexType": "INVERTED", "name": "idx_name", "fieldNames": [["name"]]}
  ]
}

Partitioning and distribution

A table takes at most one partitioning transform, either range on one column or list on one or more columns. Composite partitioning and nested fields are rejected, and every partition column must be a column of the table. Partitions can be pre-assigned in the create request, and after creation they are listed, added, and dropped through the partitions API.

The distribution is required and must be hash or even.

Gravitino distribution	Doris clause	Notes
`hash`	`DISTRIBUTED BY HASH({columns})`	Every column must exist in the table
`even`	`DISTRIBUTED BY RANDOM`	Takes no columns

The bucket number controls the BUCKETS clause. 0 omits it, so Doris applies its default, and -1 writes BUCKETS AUTO, which needs Doris 1.2.2 or later.

Hash distribution with automatic buckets
{
  "distribution": {
    "strategy": "hash",
    "number": -1,
    "funcArgs": [{"type": "field", "fieldName": ["id"]}]
  }
}

Sort orders are not supported.

Auto increment

An auto-increment column is written as AUTO_INCREMENT and needs Doris 2.1.0 or later. Gravitino reads the version from SHOW FRONTENDS when the create carries an auto-increment column, so the connecting user needs permission to run it.

A table can have only one auto-increment column, and it must be covered by a PRIMARY_KEY or UNIQUE_KEY index. Doris adds its own rules: the column must be BIGINT NOT NULL with no default.

Define an auto-increment unique key
{
  "columns": [
    {"name": "id", "type": "long", "nullable": false, "autoIncrement": true}
  ],
  "indexes": [
    {"indexType": "UNIQUE_KEY", "name": "uk_id", "fieldNames": [["id"]]}
  ]
}

Supported changes

All changes in one request other than a rename are issued as a single ALTER TABLE statement.

Change	Supported
Rename table	Yes, alone in its request
Update table comment	Yes
Add column	Yes, at any position; a default value is not applied
Delete column	Yes
Rename column	No
Update column type	Yes; the column's default is removed
Update column position	Yes
Update column comment	Yes
Update column nullable	Yes, except to `NOT NULL` on a column whose default is null
Update column default	No
Update column auto increment	No
Add index	Yes, for `INVERTED`, `BITMAP`, and `VECTOR`
Delete index	Yes
Set property	Yes
Remove property	No

Nested column names are rejected by every column change.

Identifier case and quoting

Names are case-sensitive in Gravitino and are passed to Doris exactly as given, never folded. Schema, table, and column names must match ^\w[\w/=-]{0,63}$: 1 to 64 characters, starting with an ASCII letter, digit, or underscore, and continuing with those or /, =, and -. Doris applies its own naming rules on top.

Gravitino quotes identifiers with backticks in generated SQL. Names returned by a list operation come back exactly as Doris stores them.

Type mapping

Gravitino reads each Doris type as the Gravitino type in its row, and creates each Gravitino type as the Doris type in its row, except where the list under the table says otherwise.

Doris type	Gravitino type
`BOOLEAN`	Boolean
`TINYINT`	Byte
`SMALLINT`	Short
`INT`	Integer
`BIGINT`	Long
`FLOAT`	Float
`DOUBLE`	Double
`DECIMAL(p,s)`	Decimal(p, s)
`DATE`, `DATEV2`	Date
`DATETIME(p)`	Timestamp without time zone, precision p
`DATETIME`	Timestamp without time zone
`CHAR(n)`	Char(n)
`VARCHAR(n)`	Varchar(n)
`STRING`, `TEXT`	String
`BINARY`, `VARBINARY`	Binary
`JSON`, `VARIANT`, `IPV4`, `IPV6`, `LARGEINT`, `BITMAP`, `HLL`	External type carrying the Doris type string
`ARRAY`, `MAP`, `STRUCT`, and any other type	External type carrying the Doris type string

An external type keeps the full type string in lowercase, such as array<int(11)>, so composite types are not resolved into Gravitino list, map, or struct types.

When Gravitino creates a column:

Date becomes datev2.
Timestamp without time zone becomes datetime(p), or datetime when no precision is set.
Varchar(n) takes n from 1 to 65533, and Char(n) takes n from 1 to 255.
Binary becomes binary.
An external type becomes the Doris type string it carries.
Any other Gravitino type is rejected, including Fixed, UUID, Time, timestamp with time zone, the interval types, and list, map, struct, and union types.

Column default values

CURRENT_TIMESTAMP is recognized as the current timestamp default in both directions. Literal defaults on numeric, decimal, date, datetime, char, varchar, and text columns are parsed into typed literals. Any other default, and any expression, is preserved as an unparsed expression. A function default other than CURRENT_TIMESTAMP is written inside parentheses.

Error mapping

Doris error codes are translated into Gravitino exceptions so a caller sees the same failure types it would from any other catalog. Doris often reports a failure as the generic code 1105 with the detail in the message; Gravitino then derives the code from the message text.

Doris error	Gravitino exception
1007 database exists	Schema already exists
1050 table already exists	Table already exists
1008, 1049 database does not exist	No such schema
1051, 1109 unknown table	No such table
1045 access denied for user	Unauthorized
1054 unknown column	No such column
1507 partition does not exist	No such partition
1517 duplicate partition name	Partition already exists
1064 syntax error at `AUTO`	Runtime error explaining that `BUCKETS AUTO` needs Doris `1.2.2` or later
Any other code with "Access denied" in the message	Connection failed

Any other error produces a generic runtime exception carrying the Doris message.

Quick start​

Catalog properties​

Driver version compatibility​

Schemas​

Tables​

Properties​

Indexes​

Partitioning and distribution​

Auto increment​

Supported changes​

Identifier case and quoting​

Type mapping​

Column default values​

Error mapping​

Related​