[SPARK-51660][CORE] Gracefully handle when MDC is not supported

### What changes were proposed in this pull request?
This improves the handling when MDC is not supported. The previous handling only works when one task runs on an executor. After this fix multiple tasks can run on an executor when MDC is not supported.

### Why are the changes needed?
In #35141 it added handling to gracefully handle when MDC is not available. It catches NoSuchFieldError, which is thrown during MDC initialization. This will work for the first task. If a second task runs it will not throw a NoSuchFieldError. Instead it throws NoClassDefFoundError. Then the job will hang indefinitely. This improves the fix so it works for all tasks instead of just the first one.

### Does this PR introduce _any_ user-facing change?

Yes, it fixes a bug. It also changes a log message to include the exception for easier debugging.

### How was this patch tested?
I tested it manually in spark-shell by manually throwing `NoSuchFieldError`. It logs this message and does not hang.
```
25/03/28 14:11:48 INFO Executor: MDC is not supported.
java.lang.NoSuchFieldError: this is a test
    at org.apache.spark.executor.Executor.liftedTree1$1(Executor.scala:118)
    at org.apache.spark.executor.Executor.mdcIsSupported$lzycompute(Executor.scala:114)
    at org.apache.spark.executor.Executor.mdcIsSupported(Executor.scala:113)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:948)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:584)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
```

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #50452 from robreeves/SPARK-51660.

Authored-by: Rob Reeves <roreeves@linkedin.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

Author: robreeves

core/src/main/scala/org/apache/spark/executor/Executor.scala

@@ -933,21 +933,17 @@ private[spark] class Executor(
   }
 
   private def setMDCForTask(taskName: String, mdc: Seq[(String, String)]): Unit = {
-    try {
+    if (Executor.mdcIsSupported) {
       mdc.foreach { case (key, value) => MDC.put(key, value) }
       // avoid overriding the takName by the user
       MDC.put(taskNameMDCKey, taskName)
-    } catch {
-      case _: NoSuchFieldError => logInfo("MDC is not supported.")
     }
   }
 
   private def cleanMDCForTask(taskName: String, mdc: Seq[(String, String)]): Unit = {
-    try {
+    if (Executor.mdcIsSupported) {
       mdc.foreach { case (key, _) => MDC.remove(key) }
       MDC.remove(taskNameMDCKey)
-    } catch {
-      case _: NoSuchFieldError => logInfo("MDC is not supported.")
     }
   }
 
@@ -1318,7 +1314,7 @@ private[spark] class Executor(
   }
 }
 
-private[spark] object Executor {
+private[spark] object Executor extends Logging {
   // This is reserved for internal use by components that need to read task properties before a
   // task is fully deserialized. When possible, the TaskContext.getLocalProperty call should be
   // used instead.
@@ -1327,6 +1323,21 @@ private[spark] object Executor {
   // Used to store executorSource, for local mode only
   var executorSourceLocalModeOnly: ExecutorSource = null
 
+  lazy val mdcIsSupported: Boolean = {
+    try {
+      // This tests if any class initialization error is thrown
+      val testKey = System.nanoTime().toString
+      MDC.put(testKey, "testValue")
+      MDC.remove(testKey)
+
+      true
+    } catch {
+      case t: Throwable =>
+        logInfo("MDC is not supported.", t)
+        false
+    }
+  }
+
   /**
    * Whether a `Throwable` thrown from a task is a fatal error. We will use this to decide whether
    * to kill the executor.