[SPARK-54449][CORE] Storage Memory off heap size should be considered only when off heap is enabled by VindhyaG · Pull Request #53383 · apache/spark

VindhyaG · 2025-12-08T09:40:00Z

What changes were proposed in this pull request?

Currently, even if spark.memory.offHeap.enabled=false
spark.memory.offHeap.size is calculated for SPARK UI and wherever the config is used. This PR changes the logic to take the value set in spark.memory.offHeap.size only when spark.memory.offHeap.enabled

Why are the changes needed?

Change the logic to get the config of spark.memory.offHeap.size by checking spark.memory.offHeap.enabled first

Does this PR introduce any user-facing change?

Yes. Spark UI will show the total of Onheap and Off heap memory only if spark.memory.offHeap.enabled = true.

How was this patch tested?

Existing UTs

Was this patch authored or co-authored using generative AI tooling?

No

…eap size value

This reverts commit 100dc43.

VindhyaG · 2025-12-16T03:28:40Z

@holdenk @pan3793 can you please help review this or help tag the relevant reviewers please ? Thanks!

qiaojizhen · 2026-01-21T09:05:52Z

I have also noticed this problem and I think it is necessary to correct it！

qiaojizhen · 2026-01-23T05:59:55Z

@VindhyaG
I have found another issue.
If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode.
But now the Driver is displaying the number which is set by spark.memory.offHeap.size.
So I think the Driver in the web ui should display zero for "Off Heap Memory".
Could u please solve this problem in this PR by the way？
Thanks!

VindhyaG · 2026-01-29T12:07:41Z

@VindhyaG I have found another issue. If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode. But now the Driver is displaying the number which is set by spark.memory.offHeap.size. So I think the Driver in the web ui should display zero for "Off Heap Memory". Could u please solve this problem in this PR by the way？ Thanks!

@qiaojizhen I did not get what you mean by Driver won't aply offheap in cluster mode. Is it not supported in cluster mode you mean?

VindhyaG · 2026-02-02T06:04:59Z

I have also noticed this problem and I think it is necessary to correct it！

@qiaojizhen Sorry I did not get what is the exact issue here? The value shown for the total memory size should get fixed with the changes done already.

Ngone51 · 2026-02-02T12:37:06Z

core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala

    bmConf.set(TEST_MEMORY, maxMem)
    bmConf.set(MEMORY_OFFHEAP_SIZE, maxMem)
+    if(maxMem > 0) {
+      bmConf.set(MEMORY_OFFHEAP_ENABLED, true)


Why does MEMORY_OFFHEAP_ENABLED depend on maxMem?

This is for the UTs wtih the tests where if maxMem set specifically greather than 0 only then set the MEMORY_OFFHEAP_ENABLED to true else keep it default false. Earlier MEMORY_OFFHEAP_ENABLED was always set to default false and only MEMORY_OFFHEAP_SIZE value was used to test all the combinations because earlier MEMORY_OFFHEAP_ENABLED true needed MEMORY_OFFHEAP_SIZE > 0 and not vice versa. MEMORY_OFFHEAP_ENABLED value did not matter at all for MEMORY_OFFHEAP_SIZE tests. We cannot do that with this change where if MEMORY_OFFHEAP_ENABLED is false maxMem is essentially zero as well.

Ngone51

The fix looks good to me overall. Could you post the UI screenshots before and after the fix to show the straightforward difference from UI?

qiaojizhen · 2026-02-03T05:51:53Z

@VindhyaG I have found another issue. If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode. But now the Driver is displaying the number which is set by spark.memory.offHeap.size. So I think the Driver in the web ui should display zero for "Off Heap Memory". Could u please solve this problem in this PR by the way？ Thanks!

@qiaojizhen I did not get what you mean by Driver won't aply offheap in cluster mode. Is it not supported in cluster mode you mean?

@VindhyaG Yes! The off-heap memory won't work for Driver in any cluster mode!
If I use the following config：

spark.driver.memory=1g 
spark.driver.memoryOverhead=1g 
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=2g

when I submit the job to YARN, the Driver container only gets 2G memory(including spark.driver.memory & spark.driver.memoryOverhead):

but the web ui still display the off-heap memory:

and the soucre code here Spark Yarn Client has proved that it will only use spark.driver.memory & spark.driver.memoryOverhead when applying the AM memory:

VindhyaG · 2026-02-03T12:02:14Z

The fix looks good to me overall. Could you post the UI screenshots before and after the fix to show the straightforward difference from UI?

@Ngone51 Before the change

After

VindhyaG · 2026-02-04T04:59:30Z

@VindhyaG I have found another issue. If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode. But now the Driver is displaying the number which is set by spark.memory.offHeap.size. So I think the Driver in the web ui should display zero for "Off Heap Memory". Could u please solve this problem in this PR by the way？ Thanks!

@qiaojizhen I did not get what you mean by Driver won't aply offheap in cluster mode. Is it not supported in cluster mode you mean?

@VindhyaG Yes! The off-heap memory won't work for Driver in any cluster mode! If I use the following config：
spark.driver.memory=1g 
spark.driver.memoryOverhead=1g 
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=2g
when I submit the job to YARN, the Driver container only gets 2G memory(including spark.driver.memory & spark.driver.memoryOverhead):

but the web ui still display the off-heap memory:

and the soucre code here Spark Yarn Client has proved that it will only use spark.driver.memory & spark.driver.memoryOverhead when applying the AM memory:

@qiaojizhen So basically, Driver will not have off heap no matter the setting. Is that correct? Should we not have something like NA/null (- in UI) will be more clear I suppose. Or should we just make it 0?

qiaojizhen · 2026-02-04T06:06:18Z

@VindhyaG I have found another issue. If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode. But now the Driver is displaying the number which is set by spark.memory.offHeap.size. So I think the Driver in the web ui should display zero for "Off Heap Memory". Could u please solve this problem in this PR by the way？ Thanks!

@qiaojizhen I did not get what you mean by Driver won't aply offheap in cluster mode. Is it not supported in cluster mode you mean?

@VindhyaG Yes! The off-heap memory won't work for Driver in any cluster mode! If I use the following config：
spark.driver.memory=1g 
spark.driver.memoryOverhead=1g 
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=2g
when I submit the job to YARN, the Driver container only gets 2G memory(including spark.driver.memory & spark.driver.memoryOverhead):
but the web ui still display the off-heap memory:
and the soucre code here Spark Yarn Client has proved that it will only use spark.driver.memory & spark.driver.memoryOverhead when applying the AM memory:
@qiaojizhen So basically, Driver will not have off heap no matter the setting. Is that correct? Should we not have something like NA/null (- in UI) will be more clear I suppose. Or should we just make it 0?

@VindhyaG Yes, in cluster mode, the Off-Heap of the Driver does not take effect regardless of whether the setting is enabled or not. I think "make it 0" is consistent with the existing logic.

VindhyaG · 2026-02-06T08:55:25Z

@qiaojizhen have done the changes for the driver Off heap values. Can you please revew the PR ?
@Ngone51 have attached the screenshots of before and after changes. Can you please review?

VindhyaG · 2026-02-06T10:03:43Z

When the off heap toggle is set to true

VindhyaG · 2026-02-21T03:30:38Z

@dongjoon-hyun can you please help point to a relevant reviewer for this PR?

holdenk

Seems reasonbale ish, although should we log a warning when off-heap is configured and disabled?

holdenk · 2026-02-23T20:06:55Z

core/src/main/scala/org/apache/spark/status/AppStatusListener.scala

+      if (event.blockManagerId.executorId == SparkContext.DRIVER_IDENTIFIER) {
+        exec.totalOffHeap = 0


Is it true the driver would never use off heap storage?

@holdenk The reference I took from https://spark.apache.org/docs/latest/configuration.html mentions only executor. @qiaojizhen can you please confirm again?

@holdenk @VindhyaG
In the cluster deployment mode, it is indeed so. In local mode, the Driver still uses off heap memory. But I think it will be ok to display zero for the Drvier off heap memory!

…nzero

github-actions bot added the CORE label Dec 8, 2025

VindhyaG changed the title ~~[SPARK-54449][SPARK-UI] : Storage Memory off heap size should be considered only when off heap is enabled~~ [SPARK-54449][CORE] : Storage Memory off heap size should be considered only when off heap is enabled Dec 8, 2025

VindhyaG force-pushed the SPARK-54449 branch from 4043a06 to f6be957 Compare December 8, 2025 09:57

SPARK-54449: Check if off heap config is enabled before getting off h…

4c49196

…eap size value

VindhyaG force-pushed the SPARK-54449 branch from f6be957 to 4c49196 Compare December 8, 2025 09:58

VindhyaG changed the title ~~[SPARK-54449][CORE] : Storage Memory off heap size should be considered only when off heap is enabled~~ [SPARK-54449][CORE] Storage Memory off heap size should be considered only when off heap is enabled Dec 8, 2025

VindhyaG added 4 commits December 8, 2025 15:58

SPARK-54449 : fix scala styling

100dc43

Revert "SPARK-54449 : fix scala styling"

4282a71

This reverts commit 100dc43.

SPARK-54449: fix scala styling

5615d1c

SPARK-54449: add flag to UT

a2f4e27

VindhyaG force-pushed the SPARK-54449 branch 2 times, most recently from 095d819 to db76a5d Compare December 9, 2025 13:05

UTs

0a79a8d

VindhyaG force-pushed the SPARK-54449 branch 2 times, most recently from dde0469 to 7f06280 Compare December 12, 2025 04:50

SPARK-54449: ut fix

c701ee2

VindhyaG force-pushed the SPARK-54449 branch from 7f06280 to c701ee2 Compare December 12, 2025 10:35

github-actions bot added the WEB UI label Dec 12, 2025

Merge branch 'apache:master' into SPARK-54449

a48fbaf

Ngone51 reviewed Feb 2, 2026

View reviewed changes

Merge branch 'apache:master' into SPARK-54449

951c31a

SPARK-54449: Add hover over message

dcd9d03

SPARK-54449: Driver should show 0 off heap no matter the settings

c13cc9a

VindhyaG force-pushed the SPARK-54449 branch from 551b749 to c13cc9a Compare February 4, 2026 10:20

VindhyaG added 2 commits February 6, 2026 09:05

SPARK-54449: UT fixes

00e7b74

Merge branch 'apache:master' into SPARK-54449

30fe113

holdenk reviewed Feb 23, 2026

View reviewed changes

VindhyaG added 7 commits February 24, 2026 16:14

SPARK-54449: Add warning if off heap size is disabled but value is no…

adf2292

…nzero

Merge branch 'apache:master' into SPARK-54449

20df98c

Merge branch 'apache:master' into SPARK-54449

df91792

Merge branch 'apache:master' into SPARK-54449

8fda937

Merge branch 'apache:master' into SPARK-54449

998d69a

Merge branch 'apache:master' into SPARK-54449

f684da4

Merge branch 'apache:master' into SPARK-54449

e050399

		if (event.blockManagerId.executorId == SparkContext.DRIVER_IDENTIFIER) {
		exec.totalOffHeap = 0

Conversation

VindhyaG commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

VindhyaG commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiaojizhen commented Jan 21, 2026

Uh oh!

qiaojizhen commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VindhyaG commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VindhyaG commented Feb 2, 2026

Uh oh!

Ngone51 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

VindhyaG Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ngone51 left a comment

Choose a reason for hiding this comment

Uh oh!

qiaojizhen commented Feb 3, 2026

Uh oh!

VindhyaG commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VindhyaG commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiaojizhen commented Feb 4, 2026

Uh oh!

VindhyaG commented Feb 6, 2026

Uh oh!

VindhyaG commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

VindhyaG commented Feb 21, 2026

Uh oh!

holdenk left a comment

Choose a reason for hiding this comment

Uh oh!

holdenk Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

VindhyaG Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiaojizhen Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

VindhyaG commented Dec 8, 2025 •

edited

Loading

VindhyaG commented Dec 16, 2025 •

edited

Loading

qiaojizhen commented Jan 23, 2026 •

edited

Loading

VindhyaG commented Jan 29, 2026 •

edited

Loading

VindhyaG Feb 3, 2026 •

edited

Loading

VindhyaG commented Feb 3, 2026 •

edited

Loading

VindhyaG commented Feb 4, 2026 •

edited

Loading

VindhyaG commented Feb 6, 2026 •

edited

Loading

VindhyaG Feb 24, 2026 •

edited

Loading