Architecture and concepts
Datastrato Enterprise manages metadata where it already lives. Rather than crawling source systems on a schedule and holding a copy, it connects to each source and reads and writes through to it. A table created through Gravitino exists in the underlying metastore immediately, and a table created directly in that metastore is visible without waiting for a scan.
The result is one addressable surface over metadata that is spread across lakehouses, metastores, relational databases, object storage, message queues, and model registries, without that surface becoming a second copy that drifts from the first.
Architecture
Interface layer. A REST API, with Java and Python clients over it, an MCP server for AI tools, and the administration UI.
Governance layer. Tags, policies, access control, and lineage, applied across every connected source rather than per source.
Object model. One representation of metadata regardless of where it comes from, so a table in Hive and a table in PostgreSQL are the same kind of object to a caller.
Connection layer. A connector per source type, translating between the object model and what that system natively speaks.
Engines reach data through this stack rather than around it. Trino, Spark, Flink, and any client speaking the Iceberg REST protocol resolve metadata through Gravitino, which is what makes governance enforceable rather than advisory.
Deployment
A deployment is one Datastrato Enterprise server tier in front of a relational metadata store, MySQL or PostgreSQL. Clients connect only to the server tier, which reaches each source in that source's own protocol. Engines then read and write data in storage directly; data does not pass through the server.
The server keeps its state in the metadata store, not on the pod. Datastrato Enterprise 1.3.0 runs one server replica; see Install.
The Iceberg REST service runs inside the server and shares its authentication and access control,
so Iceberg engines need no separate catalog tier. The Lance REST service
runs inside the server in the same way and serves Lance clients such as pylance, lance-spark and
lance-ray.
Terminology
| Term | Meaning |
|---|---|
| Metalake | The top-level container. Usually one per organization or tenant, holding all its catalogs |
| Catalog | A connection to one metadata source, with a connector appropriate to that source |
| Schema | The second namespace level, mapping to a database or schema in relational sources and to a logical grouping elsewhere |
| Table | A relational table in a catalog that supports them, including Iceberg and Lance tables |
| View | A stored query definition |
| Fileset | A collection of files and directories, for data that is not tabular |
| Topic | A message queue topic, in catalogs that support them |
| Model | An ML model and its versions, in catalogs that support them |
| Function | A user-defined function, exposed to engines that can call it |
Names resolve as catalog.schema.object within a metalake.
Federation
A catalog can front another catalog rather than a storage system directly. An Iceberg REST catalog in one region can proxy to one in another, so callers see a single namespace across regions or clouds without data being copied between them.
The same applies to catalogs you do not own. A Glue catalog or a vendor's Iceberg REST endpoint can be connected alongside your own, and governed by the same tags and policies.
A remote catalog is registered by name and URI, not copied, so there is no second source of truth and no replication lag. When an Iceberg engine reaches a remote Iceberg REST catalog through a local one, the local catalog forwards the caller's own token, and the remote catalog decides with its own access control, vends its own credentials and keeps its own audit log, so the data owner's controls are never delegated. The local catalog can also check the caller's privileges first; see Authorize federated access.
The life of a request
Every request, whether from an analyst, a scheduled job or an AI agent, takes the same path.
- Authenticate. The client presents a user name and password, or an OAuth token from the enterprise identity provider. Service accounts and agents authenticate the same way people do, as their own principals.
- Resolve. Datastrato Enterprise resolves
catalog.schema.objectwithin the metalake to the catalog that holds it. - Authorize. Datastrato Enterprise checks the caller's roles, privileges and ownership against the object. If it denies the request, the request ends there and never reaches the source.
- Vend credentials. For an Iceberg table read through the Iceberg REST service, or a fileset,
Datastrato Enterprise returns storage credentials scoped to that object's location. With a token
provider such as
s3-tokenthey are short-lived, one hour by default; see Credential vending. - Execute. The engine reads or writes the data in storage directly. Data does not pass through Datastrato Enterprise and is not copied.
- Audit. The server writes an audit record of the user, the operation, the object, the time and the outcome.
Multiple engines
Gravitino does not process queries. Engines do, and they reach it in one of two ways.
A native connector, for Trino, Spark, Flink, and Daft, loads catalogs from Gravitino directly and gives the engine the full object model.
The Iceberg REST protocol serves anything that speaks it, including Doris, StarRocks, PyIceberg, and Ray, without a connector.
Either path enforces the same access control, so a policy written once applies regardless of which engine a user reaches the data through.
Lance clients such as pylance, lance-spark and lance-ray reach Lance tables
through the Lance REST service, which tells them where each table lives;
they then read and write the Lance data in storage directly.
Data and AI assets
Filesets and models are first-class objects rather than an afterthought. Training data, feature sets, and model versions are cataloged, tagged, and governed the same way tables are, which matters when the same governance obligations apply to a training corpus as to a warehouse.
Lance tables hold the output of AI pipelines: passages from documents, their embedding vectors, and pointers to the files they came from, searchable by meaning. Source files in a fileset and the Lance table built from them are cataloged side by side. See How Lance works for the format and for a pipeline from files to a searchable table, and How Iceberg works for the table format behind most analytics tables.