HiveQL compatibility

A hive job runs your HiveQL script with Spark SQL 3.5, in a single pod, on the tenant’s Apache Iceberg catalog. There is no HiveServer2, no Tez and no Thrift metastore to operate: the tables the script creates are Iceberg tables on your object storage, readable afterwards by Trino, Spark and the other jobs of the project.

This page says what runs as is, what gets rewritten, and how to take over the tables that still live in a Hive metastore. The compatibility lists come from the Spark and Iceberg documentation, cited and dated at the bottom of the page.

How the script runs

Step	What happens
Default catalog	`graal`, the tenant’s Iceberg catalog
Current database	the project’s database (`USE graal.<database>` is set before your script)
Engine	`spark-sql` in local mode, cores and memory of the chosen instance type
Configuration	the job’s `conf` keys are passed as `--conf`; those that make no sense in one pod are dropped
Log and state	the run log carries the `spark-sql` output; the state comes from the exit code

What runs as is

Spark SQL covers most of Hive’s query language:

SELECT, GROUP BY, ORDER BY, DISTRIBUTE BY, CLUSTER BY, SORT BY;
relational, arithmetic and logical operators, complex type constructors, math and string functions;
joins (JOIN, outer joins, LEFT SEMI JOIN, LEFT ANTI JOIN, CROSS JOIN), UNION, sub-queries in FROM, WHERE (IN, EXISTS) and join conditions;
window functions, sampling, EXPLAIN;
views (with explicit column aliases for a view created by Hive);
the TINYINT to BIGINT, FLOAT, DOUBLE, DECIMAL, STRING, BINARY, BOOLEAN, TIMESTAMP, DATE, ARRAY<>, MAP<>, STRUCT<> types;
CREATE TABLE … AS SELECT, INSERT INTO, INSERT OVERWRITE, ALTER TABLE.

What gets rewritten

Tables are Iceberg tables: the “storage” part of Hive DDL changes shape.

Original HiveQL	In graal
`STORED AS ORC` / `PARQUET`, `ROW FORMAT SERDE`	`USING iceberg`, format chosen by `TBLPROPERTIES ('write.format.default'='orc')`
`PARTITIONED BY (dt STRING)`	`PARTITIONED BY (dt)` on a column of the table, or a transform: `days(ts)`, `months(ts)`
`CLUSTERED BY (id) INTO 16 BUCKETS`	`PARTITIONED BY (bucket(16, id))`
`LOCATION 'hdfs://…'`	nothing to write: the table lives in the catalog’s object storage; HDFS files are copied before
`CREATE TABLE … LIKE`	`CREATE TABLE … AS SELECT … LIMIT 0`, or an explicit DDL
`SET hive.exec.`, `SET mapreduce.`	no effect: Spark reads neither Hive nor MapReduce settings

INSERT OVERWRITE on a partitioned table follows Spark’s mode, static by default: without a PARTITION clause, the whole table is replaced. To get Hive’s behaviour back (only the partitions the query produces are replaced), start the script with SET spark.sql.sources.partitionOverwriteMode=dynamic;, the mode Iceberg recommends. Iceberg’s partition transforms (years, months, days, hours, bucket, truncate) replace partition columns computed by hand.

Behaviour differences to know

These Hive functions exist in Spark SQL but return a different result:

Expression	Hive	Spark SQL
`SQRT(n)` with `n < 0`	`NULL`	`NaN`
`ACOS(n)`, `ASIN(n)` outside `[-1, 1]`	`NULL`	`NaN`
`CAST(n AS TIMESTAMP)`, integer `n`	`n` in milliseconds	`n` in seconds

And these Hive features have no equivalent in Spark SQL: the UNIONTYPE type, the UNIQUEJOIN clause, bitmap indexes and virtual columns, the STREAMTABLE join hint, reading Hadoop archives (HAR). Hive-specific optimizations (number of reducers, merging small output files, skew flag) are replaced by Spark’s. A Java function written for Hive (UDF, UDAF, UDTF) can be called from Spark SQL, except for a few methods of the Hive API that Spark does not call (getRequiredJars, getRequiredFiles, configure, close, reset, getWindowingEvaluator).

Taking over the tables of a Hive metastore

Need	What to use
Read existing tables in place	a `hive_metastore` data connection, cited by the job
Copy HDFS files to S3	the `hdfs-copy` job type, which checks files and bytes after the copy
Convert a Hive table to an Iceberg table	the `iceberg-migration` job type (`snapshot`, `migrate`, `add_files`)
Inventory a cluster before migrating	the `hadoop-assess` job type, which produces a report of the estate

Sources

Apache Spark 3.5.6, “Compatibility with Apache Hive”, accessed on 29 October 2026.
Apache Iceberg 1.10.0, “Spark DDL”, accessed on 29 October 2026.