feat: Add config to enable running Comet in onheap mode

[andygrove]

Which issue does this PR close?

Related to #2342

Rationale for this change

Running Comet without Spark's off-heap mode enabled is not recommended for production use, as documented in the tuning guide. We should fall back to Spark by default in on-heap mode unless the user explicitly opts in.

What changes are included in this PR?

• New config
• Fall back to Spark if config is not enabled
• Update GitHub workflows for Spark SQL test to enable new config via env var
• Update CometTestBase to enable new config
• Update docs

How are these changes tested?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.89%. Comparing base (f09f8af) to head (72a8664).
⚠️ Report is 1150 commits behind head on main.

Files with missing lines	Patch %	Lines
...park/src/main/scala/org/apache/spark/Plugins.scala	14.28%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2554      +/-   ##
============================================
+ Coverage     56.12%   58.89%   +2.77%     
- Complexity      976     1457     +481     
============================================
  Files           119      147      +28     
  Lines         11743    13652    +1909     
  Branches       2251     2371     +120     
============================================
+ Hits           6591     8041    +1450     
- Misses         4012     4386     +374     
- Partials       1140     1225      +85     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wForget]

Thanks, lgtm

[andygrove]

@Kontinuation @EmilyMatt fyi

[parthchandra]

Why do we need a new config for this? Isn't Spark's config sufficient?

[andygrove]

Why do we need a new config for this? Isn't Spark's config sufficient?

The goal is to require the user to opt-in to enabling Comet if Spark is in on-heap mode, because running Comet in on-heap mode is not recommended for production use.

[parthchandra]

Why do we need a new config for this? Isn't Spark's config sufficient?

The goal is to require the user to opt-in to enabling Comet if Spark is in on-heap mode, because running Comet in on-heap mode is not recommended for production use.

Hmm. I feel that we have too many configs already and there may already be confusion about memory settings for Comet. I'm not saying we shouldn't have this config, but we do need to start simplifying things for end users soon.

[andygrove]

Why do we need a new config for this? Isn't Spark's config sufficient?

The goal is to require the user to opt-in to enabling Comet if Spark is in on-heap mode, because running Comet in on-heap mode is not recommended for production use.

Hmm. I feel that we have too many configs already and there may already be confusion about memory settings for Comet. I'm not saying we shouldn't have this config, but we do need to start simplifying things for end users soon.

I agree. I think you will like the next PR that this enables. Thanks for the review.