Data and SQL

The principle fits in one sentence: your files stay on your object storage. graal Data Platform catalogs them and makes them queryable; it does not host them and does not copy them elsewhere.

Buckets and explorer

Each project has buckets on the S3-compatible storage you provide. The explorer serves the two everyday gestures: dropping a dataset, and looking at what a run has just produced.

Jobs write there in Parquet by default. It is an open format, readable by Spark, Pandas, DuckDB and Trino without conversion.

Catalog: layers, databases, tables, fields

The catalog is organised in four levels:

layer — the refining stage (raw, cleaned, exposed, per your own convention);
database — the logical grouping inside a layer;
table — the dataset itself, backed by files in the bucket;
field — the columns, their type and their description.

It is not just a directory: it is what makes it possible to know what you have and who may touch it, project by project. Knowledge of the data does not stay in the head of whoever dropped it.

Rights are set per layer, per database and per table, with the same roles and assignments as the rest of the platform (see roles and permissions).

Apache Iceberg tables

Tables use the Apache Iceberg format, on your object storage: transactions, schema evolution and time travel, in a format the whole ecosystem reads.

The Iceberg catalog is a standard REST catalog, backed by the Data Platform’s PostgreSQL database, with one namespace per tenant. Trino, Spark, Flink, Dask and PyIceberg all read it, by table name. When a job writes a table, its fields appear in the catalog with no manual entry. The details are in The data catalog.

Read and write a table by name from a job

A job gets its tenant’s graal Iceberg catalog with nothing to configure: no S3 path, no key, no catalog address. The run carries a token valid for that run only, and storage access is signed by the platform. The default schema is the tenant’s.

# Spark (spark job, PySpark)
df = spark.table("sales")
df.groupBy("region").count().writeTo("sales_by_region").append()

# Python, notebook, Ray: the graal module is provided to the run
import graal
df = graal.read_table("sales")                   # pandas
graal.write_table(df, "sales_clean")             # creates the table on first call

# Dask: one partition per file, read by the workers
ddf = graal.read_table_dask("sales")

// Flink: the catalog is provided to the run as SQL
tEnv.executeSql(Files.readString(Path.of(System.getenv("GRAAL_FLINK_CATALOG_SQL"))));
tEnv.useCatalog("graal");
tEnv.executeSql("SELECT count(*) FROM sales").print();

SQL warehouse: Trino

The warehouse queries your tables in SQL with Trino, deployed by the platform chart in the application namespace. A catalog table becomes queryable with no copy and no prior loading, and Trino joins your Iceberg tables with your existing databases.

Queries run from the console’s SQL editor, from a notebook, or from your BI tool through Trino’s standard JDBC access. Existing Hive-format Parquet or ORC tables are read on the same catalog.

Lineage

The catalog tells where each table comes from: lineage is followed at column level, from the job that produced it to the sources it read. The site’s Lakehouse and SQL page also presents AI-generated dataset documentation.