When using Trino to optimize Iceberg tables, my table data exceeded 5 billion rows. Since Iceberg's optimization feature does not enable disk swapping, the optimization fails due to insufficient memory. Is there a solution?

[c8679724]

When using Trino to optimize Iceberg tables, my table data exceeded 5 billion rows. Since Iceberg's optimization feature does not enable disk swapping, the optimization fails due to insufficient memory. Is there a solution?

I noticed that Spark also has optimization failures when there are many small files in Iceberg.

[mosabua]

You can just optimize parts of the table only .. either using partitions specifically or with a where condition - https://trino.io/docs/current/connector/iceberg.html#optimize

[c8679724]

Currently, I see that many syntaxes in Trino support spilling to disk. Is there any plan for optimization in this regard?

[mosabua]

We have fault tolerant execution mode and it will transparently manage this. Spilling to disc wont help and it unrelated.