[SPARK-49388] Remove SPARK_NO_DAEMONIZE in favor of live log UIs

### What changes were proposed in this pull request?

This PR aims to remove `SPARK_NO_DAEMONIZE` in favor of live log UIs.
- apache/spark#44890
- apache/spark#44888

### Why are the changes needed?

Spark Master/Worker supports live log UIs already. A user doesn't need to use `kubectl` to access K8s log.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#103 from dongjoon-hyun/SPARK-49388.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

Author: dongjoon-hyun

spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkClusterResourceSpec.java

@@ -176,15 +176,14 @@ private static StatefulSet buildMasterStatefulSet(
         .withName("master")
         .withImage(image)
         .addNewEnv()
-        .withName("SPARK_NO_DAEMONIZE")
-        .withValue("1")
-        .endEnv()
-        .addNewEnv()
         .withName("SPARK_MASTER_OPTS")
         .withValue(options)
         .endEnv()
         .addToCommand("bash")
-        .addToArgs("/opt/spark/sbin/start-master.sh")
+        .addToArgs(
+            "-c",
+            "/opt/spark/sbin/start-master.sh && while /opt/spark/sbin/spark-daemon.sh status "
+                + "org.apache.spark.deploy.master.Master 1; do sleep 1; done")
         .addNewPort()
         .withName("web")
         .withContainerPort(8080)
@@ -240,15 +239,20 @@ private static StatefulSet buildWorkerStatefulSet(
         .withName("worker")
         .withImage(image)
         .addNewEnv()
-        .withName("SPARK_NO_DAEMONIZE")
-        .withValue("1")
+        .withName("SPARK_LOG_DIR")
+        .withValue("/opt/spark/work/logs")
         .endEnv()
         .addNewEnv()
         .withName("SPARK_WORKER_OPTS")
         .withValue(options)
         .endEnv()
         .addToCommand("bash")
-        .addToArgs("/opt/spark/sbin/start-worker.sh", "spark://" + name + "-master-svc:7077")
+        .addToArgs(
+            "-c",
+            "/opt/spark/sbin/start-worker.sh spark://"
+                + name
+                + "-master-svc:7077 && while /opt/spark/sbin/spark-daemon.sh status "
+                + "org.apache.spark.deploy.worker.Worker 1; do sleep 1; done")
         .endContainer()
         .endSpec()
         .endTemplate()