Skip to content

Commit 24e6c18

Browse files
pgandhiTom Graves
authored andcommitted
[SPARK-21798] No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server
History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server. Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode. ## How was this patch tested? Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running. Author: pgandhi <[email protected]> Author: pgandhi999 <[email protected]> Closes apache#19047 from pgandhi999/master.
1 parent 0456b40 commit 24e6c18

File tree

5 files changed

+16
-0
lines changed

5 files changed

+16
-0
lines changed

conf/spark-env.sh.template

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@
5252
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
5353
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
5454
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
55+
# - SPARK_DAEMON_CLASSPATH, to set the classpath for all daemons
5556
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers
5657

5758
# Generic options for the daemons used in the standalone deploy mode

docs/monitoring.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ The history server can be configured as follows:
6161
<td><code>SPARK_DAEMON_JAVA_OPTS</code></td>
6262
<td>JVM options for the history server (default: none).</td>
6363
</tr>
64+
<tr>
65+
<td><code>SPARK_DAEMON_CLASSPATH</code></td>
66+
<td>Classpath for the history server (default: none).</td>
67+
</tr>
6468
<tr>
6569
<td><code>SPARK_PUBLIC_DNS</code></td>
6670
<td>

docs/running-on-mesos.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,8 @@ If you like to run the `MesosClusterDispatcher` with Marathon, you need to run t
160160
The `MesosClusterDispatcher` also supports writing recovery state into Zookeeper. This will allow the `MesosClusterDispatcher` to be able to recover all submitted and running containers on relaunch. In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env by configuring `spark.deploy.recoveryMode` and related spark.deploy.zookeeper.* configurations.
161161
For more information about these configurations please refer to the configurations [doc](configurations.html#deploy).
162162

163+
You can also specify any additional jars required by the `MesosClusterDispatcher` in the classpath by setting the environment variable SPARK_DAEMON_CLASSPATH in spark-env.
164+
163165
From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying the master URL
164166
to the URL of the `MesosClusterDispatcher` (e.g: mesos://dispatcher:7077). You can view driver statuses on the
165167
Spark cluster Web UI.

docs/spark-standalone.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,10 @@ You can optionally configure the cluster further by setting environment variable
149149
<td><code>SPARK_DAEMON_JAVA_OPTS</code></td>
150150
<td>JVM options for the Spark master and worker daemons themselves in the form "-Dx=y" (default: none).</td>
151151
</tr>
152+
<tr>
153+
<td><code>SPARK_DAEMON_CLASSPATH</code></td>
154+
<td>Classpath for the Spark master and worker daemons themselves (default: none).</td>
155+
</tr>
152156
<tr>
153157
<td><code>SPARK_PUBLIC_DNS</code></td>
154158
<td>The public DNS name of the Spark master and workers (default: none).</td>

launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,16 +53,19 @@ public List<String> buildCommand(Map<String, String> env)
5353
case "org.apache.spark.deploy.master.Master":
5454
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
5555
javaOptsKeys.add("SPARK_MASTER_OPTS");
56+
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
5657
memKey = "SPARK_DAEMON_MEMORY";
5758
break;
5859
case "org.apache.spark.deploy.worker.Worker":
5960
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
6061
javaOptsKeys.add("SPARK_WORKER_OPTS");
62+
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
6163
memKey = "SPARK_DAEMON_MEMORY";
6264
break;
6365
case "org.apache.spark.deploy.history.HistoryServer":
6466
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
6567
javaOptsKeys.add("SPARK_HISTORY_OPTS");
68+
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
6669
memKey = "SPARK_DAEMON_MEMORY";
6770
break;
6871
case "org.apache.spark.executor.CoarseGrainedExecutorBackend":
@@ -77,11 +80,13 @@ public List<String> buildCommand(Map<String, String> env)
7780
break;
7881
case "org.apache.spark.deploy.mesos.MesosClusterDispatcher":
7982
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
83+
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
8084
break;
8185
case "org.apache.spark.deploy.ExternalShuffleService":
8286
case "org.apache.spark.deploy.mesos.MesosExternalShuffleService":
8387
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
8488
javaOptsKeys.add("SPARK_SHUFFLE_OPTS");
89+
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
8590
memKey = "SPARK_DAEMON_MEMORY";
8691
break;
8792
default:

0 commit comments

Comments
 (0)