Gluten + Velox fallback when reading Delta Lake (Spark 3.5.3 + Delta 3.3.2)

[vardaram-zetta]

I am using Spark 3.5.3, Delta Lake 3.3.2, and the latest Gluten + Velox build.
My TPC-H 2GB tables were written as Delta on S3A using regular Spark.
When I query the Delta tables with Gluten enabled, execution falls back.

Example:

select sum(p_partkey) from part;

Fallback Messages

25/12/04 17:38:43 WARN GlutenFallbackReporter: Validation failed for plan: Scan json [QueryId=1], due to:
 - Unsupported file format UnknownFormat.
25/12/04 17:38:48 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
25/12/04 17:38:48 WARN GlutenFallbackReporter: Validation failed for plan: Scan json [QueryId=3], due to:
 - Unsupported file format UnknownFormat.
25/12/04 17:38:48 WARN GlutenFallbackReporter: Validation failed for plan: Project[QueryId=3], due to:
 - Validation failed with exception from: ProjectExecTransformer, reason: UDF name is not found!

Question

Does Gluten + Velox fully support Delta Lake 3.x on Spark 3.5.x?
Or is fallback expected with this combination?

If there is a specific Delta version recommended for Spark 3.5 + Gluten + Velox, please let me know.

Environment

• Spark 3.5.3
• Delta Lake 3.3.2
• Latest Gluten (built from main)
• Velox backend
• Data stored on S3A (Delta format)

[zhztheplayer]

25/12/04 17:38:43 WARN GlutenFallbackReporter: Validation failed for plan: Scan json [QueryId=1], due to:

• Unsupported file format UnknownFormat.

This looks normal as we haven't offloaded the JSON scan used in Delta Lake read yet. They are for querying Delta Lake's metadata and we don't expect them to cause performance overhead since they are less heavier than the actual Parquet data scan.

25/12/04 17:38:48 WARN GlutenFallbackReporter: Validation failed for plan: Project[QueryId=3], due to:

• Validation failed with exception from: ProjectExecTransformer, reason: UDF name is not found!

Where is this project from? I assume it's not from the Parquet read too, which is also fine. Would you share the query plan for this one?

[vardaram-zetta]

Hi @zhztheplayer , thanks for the response,
Here is the detailed plan for the query select sum(p_partkey) from part;
On the tab Gluten SQL/Dataframe we have a total of 4 executions ids as follows

[zhztheplayer]

These query plans look as expected. The Parquet scan is offloaded, the others are for querying Delta metadata and we haven't yet fully offloaded them.

Any performance issue observed around the metadata queries?

[vardaram-zetta]

@zhztheplayer I haven’t noticed any performance issues so far, but I also haven’t tested heavy workloads yet — only a 10GB TPCH dataset for basic compatibility checks.

Also, does Gluten + Velox currently support Delta Lake Deletion Vectors, or do those still fall back to Spark’s native operators?

[zhztheplayer]

For deletion vectors, we have a pending PR #10740 for Spark 3.4 / Delta 2.4. It's a PoC. And it's not difficult to port it to Spark 3.5 / Delta 3.3 once landed.

[vardaram-zetta]

@zhztheplayer thank you so much!