[Enhancement] Support recursive spill disk operator

[yiguolei]

Search before asking

• I had searched in the issues and found no similar issues.

Description

At present, Doris's disk spilling mechanism works by splitting a hash table into a fixed number of partitions (e.g., 16) for disk writing, with no recursion implemented. Therefore, when memory is truly insufficient, the actual maximum memory footprint for disk spilling is only 15/16 of the total. We need to implement a recursive strategy that allows further recursive partitioning within each 1/16 partition. The goal is to run all TPC-DS 1TB queries with just 10GB of memory.

Solution

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct