[SPARK-55353][SQL] Disable SQLAppStatusListener when UI is disabled by ever4Kenny · Pull Request #54133 · apache/spark

ever4Kenny · 2026-02-04T06:30:11Z

What changes were proposed in this pull request?

This patch modifies SharedState to skip SQLAppStatusListener creation when Spark UI is disabled (spark.ui.enabled=false).

Why are the changes needed?

Complex SQL queries with many stages and tasks cause driver OOM due to excessive AccumulatorMetadata objects held by SQLAppStatusListener.

Memory regression from Spark 3.2 to 3.5:

Accumulator count per task increased 4x due to new metrics (SPARK-36620, SPARK-40711, SPARK-43214)
Example: 155 stages × 1000 tasks × 100 accumulators = 15.5M objects (~1.5GB driver memory)
Same query runs on 3.2.2 with 2GB driver but OOMs on 3.5.7 with 8GB driver

By reusing the existing spark.ui.enabled configuration, users can disable UI to reduce memory overhead without needing a separate configuration.

Does this PR introduce any user-facing change?

No new configuration is added. The behavior change is:

When spark.ui.enabled=false:

SQLAppStatusListener is not created, significantly reducing driver memory usage
No SQL tab in live Spark UI (expected since UI is disabled)
Event logs are still written for History Server analysis

How was this patch tested?

Added unit tests in SharedStateSuite to verify SQLAppStatusListener creation behavior
Tested with production workload: OOM resolved when spark.ui.enabled=false

Was this patch authored or co-authored using generative AI tooling?

No.

github-actions · 2026-02-04T06:30:19Z

JIRA Issue Information

=== Improvement SPARK-55353 ===
Summary: Driver OOM regression with complex SQL queries: Add config to disable SQLAppStatusListener
Assignee: None
Status: Open
Affected: ["3.5.7"]

This comment was automatically generated by GitHub Actions

pan3793 · 2026-02-06T03:20:54Z

SQLAppStatusListener is not created, significantly reducing driver memory usage

How much memory decreased in your case?

Limited SQL execution metrics in live Spark UI

Do you have a concrete list of the impact?

ever4Kenny · 2026-02-10T03:36:48Z

SQLAppStatusListener is not created, significantly reducing driver memory usage

How much memory decreased in your case?

Limited SQL execution metrics in live Spark UI

Do you have a concrete list of the impact?

At least 6GB memory were decreased(with listener 8G driver exited with oom, w/o listener 2G driver ran fine)
In my case I hardly notice any change, the SQL tab still existed, the DAG and the metrics on operators were still intact both in live UI or history UI

pan3793 · 2026-02-10T10:14:40Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

+        "many stages and tasks, but will result in limited SQL execution information in the UI.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)


Spark currently under 4.2 development cycle so version should be "4.2.0".

I agree appStatusListener should keep enabled by default. But could you disable it and tigger a round CI, to help us evaluate the impact?

Ok, I will try disable it and trigger a CI, and enable it afterwards.

Few metrics related tests failed with the listener disabled, could you please take a look?

mridulm · 2026-02-16T03:27:09Z

sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala

+    } else {
+      // When disabled, create a minimal status store without listener
+      new SQLAppStatusStore(kvStore, None)
+    }


If this is specific to UI, control it using spark.ui.enabled and avoid introducing a config ? Similar (as a pattern) to streamingQueryStatusListener below.

Good point, I will use "spark.ui.enabled" instead.

I'm a little confused, I believe the author means: UI almost works even if appStatusListener is disabled.

If so, separate configs make sense.

I'm a little confused, I believe the author means: UI almost works even if appStatusListener is disabled.

If so, separate configs make sense.

Yes that's true, the UI is still available without the listener.

the impacts looks fine! (you may need to rebase your branch to the latest master to solve the Python failures)

So how about

mark the new config spark.sql.ui.appStatusListener.enabled as internal, to make it for advanced users only, as it has side effects that might confuse normal users.

keep the default value fallback to spark.ui.enabled, then disabling UI also reduces driver memory usage.

@mridulm @cloud-fan, WDYT? Does disabling appStatusListener have other potential impacts other than UI?

Thanks for your help!

cloud-fan · 2026-02-25T09:31:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

@@ -236,6 +236,15 @@ object StaticSQLConf {
      .booleanConf
      .createWithDefault(false)

+  val SQL_UI_APP_STATUS_LISTENER_ENABLED =
+    buildStaticConf("spark.sql.ui.appStatusListener.enabled")


appStatusListener is internal, we should not expose it to users. So this is to disable SQL UI?

Ok, directly use spark.ui.enabled to toggle the lisener.

I believe what @cloud-fan said "disable SQL UI" means "disable SQLTab", not the whole live UI.

You probably need a new config like spark.sql.ui.enabled

When spark.ui.enabled is set to false, SQLAppStatusListener should not be created to avoid unnecessary memory overhead. This can significantly reduce driver memory usage for complex queries with many stages and tasks. The change checks spark.ui.enabled configuration value instead of creating a separate configuration, reusing the existing spark.ui.enabled setting to control SQLAppStatusListener lifecycle.

github-actions bot added the SQL label Feb 4, 2026

ever4Kenny force-pushed the SPARK-55353-disable-sql-app-status-listener branch from c44cf81 to 7086200 Compare February 4, 2026 06:40

pan3793 reviewed Feb 10, 2026

View reviewed changes

mridulm reviewed Feb 16, 2026

View reviewed changes

ever4Kenny force-pushed the SPARK-55353-disable-sql-app-status-listener branch 2 times, most recently from 0ee4e60 to f92f2b7 Compare February 25, 2026 02:57

cloud-fan reviewed Feb 25, 2026

View reviewed changes

ever4Kenny force-pushed the SPARK-55353-disable-sql-app-status-listener branch from f92f2b7 to d121348 Compare February 25, 2026 10:18

ever4Kenny changed the title ~~[SPARK-55353][SQL] Add config to disable SQLAppStatusListener~~ [SPARK-55353][SQL] Disable SQLAppStatusListener when UI is disabled Feb 25, 2026

ever4Kenny force-pushed the SPARK-55353-disable-sql-app-status-listener branch 2 times, most recently from ff5aeae to 9281179 Compare February 26, 2026 03:31

ever4Kenny force-pushed the SPARK-55353-disable-sql-app-status-listener branch from 9281179 to 5d31b42 Compare February 26, 2026 07:21

Conversation

ever4Kenny commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions bot commented Feb 4, 2026

JIRA Issue Information

Uh oh!

pan3793 commented Feb 6, 2026

Uh oh!

ever4Kenny commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ever4Kenny Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mridulm Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ever4Kenny commented Feb 4, 2026 •

edited

Loading

ever4Kenny commented Feb 10, 2026 •

edited

Loading

pan3793 Feb 10, 2026 •

edited

Loading

ever4Kenny Feb 25, 2026 •

edited

Loading

mridulm Feb 16, 2026 •

edited

Loading

pan3793 Feb 25, 2026 •

edited

Loading