Datafusion with IcebergTableScan's performance is much slower than DeltaLake

[Smith-Cruise]

Is your feature request related to a problem or challenge?

I'm using DataFusion to integrate iceberg-rust and delta-rs. However, I found that the performance of iceberg-rust is not ideal.

I've built a simple benchmark on my machine, TPCH SF100 parquet lineitem fully scan.

select * from catalog.tpch_sf100.lineitem order by 1 limit 1;

iceberg-rust: 104s
delta-rs: 40s

In the iceberg-rust, it looks like we can't parallel scan. The CPU's utilization is very low. Sometimes, only one core is 100%.

Could anyone help me find the root cause?

Describe the solution you'd like

No response

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community

[CTTY]

This is a known limitation. The existing scan doesn't implement output partitioning: https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/physical_plan/scan.rs#L105

[Smith-Cruise]

This is a known limitation. The existing scan doesn't implement output partitioning: https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/physical_plan/scan.rs#L105

But I've seen there is a parallel handle FileScanTask in IcebergTableScan, is that parallel didn't work?

https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/arrow/reader.rs#L150

[liurenjie1024]

This is a known limitation. The existing scan doesn't implement output partitioning: https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/physical_plan/scan.rs#L105

But I've seen there is a parallel handle FileScanTask in IcebergTableScan, is that parallel didn't work?

https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/arrow/reader.rs#L150

It's in fact io parallelism, not cpu parallelism.