Description

GLUTEN-8018 increases heap memory when the fallback ratio over spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold may cause high memory consumption. For example, 10G for heap memory, and 15G for off-heap memory, using the follow setting.

'
spark.gluten.auto.adjustStageResource.enabled=true;
spark.gluten.auto.adjustStageResources.heap.ratio=3.0d;
spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold=0.1d;
'

When this rule is applied, heap memory will become 30G, and the total memory for gluten is 45G, that brings 20G consumption and causes OOM.

In my ideal, it's better to decrease off-heap memory at the same time when increasing heap memory to make the total memory consumption no-increasing.

Gluten version

None