feat: Support comet native log level conf

[wForget]

Which issue does this PR close?

Closes #2309

Rationale for this change

Makes it easier for us to modify the native log level

What changes are included in this PR?

Add COMET_LOG_LEVEL env to allow specifying native log level

How are these changes tested?

Adding COMET_LOG_LEVEL=DEBUG overrides default log level:

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.50000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.44%. Comparing base (f09f8af) to head (597b362).
⚠️ Report is 501 commits behind head on main.

Files with missing lines	Patch %	Lines
...mon/src/main/java/org/apache/comet/NativeBase.java	40.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2379      +/-   ##
============================================
+ Coverage     56.12%   57.44%   +1.31%     
- Complexity      976     1297     +321     
============================================
  Files           119      147      +28     
  Lines         11743    13424    +1681     
  Branches       2251     2351     +100     
============================================
+ Hits           6591     7711    +1120     
- Misses         4012     4451     +439     
- Partials       1140     1262     +122     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

@wForget Could you a note to the debugging guide?

[wForget]

@wForget Could you a note to the debugging guide?

Sure. Also, can I change these log properties to environments? It seems easier to configure them in spark using something like like spark.executorEnv.XXX?

[wForget]

Also, can I change these log properties to environments? It seems easier to configure them in spark using something like like spark.executorEnv.XXX?

Previously, the COMET_CONF_DIR environment variable was used to specify the log configuration file dir. I added a COMET_LOG_LEVEL environment variable to specify the log level.

[wForget]

@andygrove Could you please take another look? I added the document.

[wForget]

Corrected to emits to stderr

datafusion-comet/native/core/src/lib.rs

Line 106 in a53e090

.target(Target::Stderr)

[parthchandra]

Why change to stderr? Shouldn't the default log level should be info, emitting to stdout?

[wForget]

Why change to stderr? Shouldn't the default log level should be info, emitting to stdout?

This is just a log fix, the behavior change was introduced by #164

[comphead]

LGTM thanks @wForget

[andygrove]

Thanks @wForget