[FEATURE] Better memory limitation when overlapping decompression is enabled

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Currently, the overlapping decompression mechanism may cause Spark jobs to run out of memory due to the lack of memory limits.

In the first phase, we can introduce limits on the number of staging buffers. Subsequently, we can align with Spark’s memory consumer model to precisely request and release memory.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

cc @wForget

[wForget]

Since #2717 introduced decompressionBytes counters, can we block decompress threads when decompressionBytes reaches a threshold?

[zuston]

Since #2717 introduced decompressionBytes counters, can we block decompress threads when decompressionBytes reaches a threshold?

#2735