Skip to content

Commit 57aff93

Browse files
gaborgsomogyiMarcelo Vanzin
authored andcommitted
[SPARK-26998][CORE] Remove SSL configuration from executors
## What changes were proposed in this pull request? Different SSL passwords shown up as command line argument on executor side in standalone mode: * keyStorePassword * keyPassword * trustStorePassword In this PR I've removed SSL configurations from executors. ## How was this patch tested? Existing + additional unit tests. Additionally tested with standalone mode and checked the command line arguments: ``` [gaborsomogyi:~/spark] SPARK-26998(+4/-0,3)+ ± jps 94803 CoarseGrainedExecutorBackend 94818 Jps 90149 RemoteMavenServer 91925 Nailgun 94793 SparkSubmit 94680 Worker 94556 Master 398 [gaborsomogyi:~/spark] SPARK-26998(+4/-1,3)+ ± ps -ef | egrep "94556|94680|94793|94803" 502 94556 1 0 2:02PM ttys007 0:07.39 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host gsomogyi-MBP.local --port 7077 --webui-port 8080 --properties-file conf/spark-defaults.conf 502 94680 1 0 2:02PM ttys007 0:07.27 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 --properties-file conf/spark-defaults.conf spark://gsomogyi-MBP.local:7077 502 94793 94782 0 2:02PM ttys007 0:35.52 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Dscala.usejavacp=true -Xmx1g org.apache.spark.deploy.SparkSubmit --master spark://gsomogyi-MBP.local:7077 --class org.apache.spark.repl.Main --name Spark shell spark-shell 502 94803 94680 0 2:03PM ttys007 0:05.20 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/java -cp /Users/gaborsomogyi/spark/conf/:/Users/gaborsomogyi/spark/assembly/target/scala-2.12/jars/* -Xmx1024M -Dspark.ssl.ui.port=0 -Dspark.driver.port=60902 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler172.30.65.186:60902 --executor-id 0 --hostname 172.30.65.186 --cores 8 --app-id app-20190326140311-0000 --worker-url spark://Worker172.30.65.186:60899 502 94910 57352 0 2:05PM ttys008 0:00.00 egrep 94556|94680|94793|94803 ``` Closes apache#24170 from gaborgsomogyi/SPARK-26998. Authored-by: Gabor Somogyi <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
1 parent d4420b4 commit 57aff93

File tree

2 files changed

+11
-1
lines changed

2 files changed

+11
-1
lines changed

core/src/main/scala/org/apache/spark/SparkConf.scala

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -738,7 +738,6 @@ private[spark] object SparkConf extends Logging {
738738
*/
739739
def isExecutorStartupConf(name: String): Boolean = {
740740
(name.startsWith("spark.auth") && name != SecurityManager.SPARK_AUTH_SECRET_CONF) ||
741-
name.startsWith("spark.ssl") ||
742741
name.startsWith("spark.rpc") ||
743742
name.startsWith("spark.network") ||
744743
isSparkPortConf(name)

core/src/test/scala/org/apache/spark/SparkConfSuite.scala

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -353,6 +353,17 @@ class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSyst
353353
}
354354
}
355355

356+
test("SPARK-26998: SSL configuration not needed on executors") {
357+
val conf = new SparkConf(false)
358+
conf.set("spark.ssl.enabled", "true")
359+
conf.set("spark.ssl.keyPassword", "password")
360+
conf.set("spark.ssl.keyStorePassword", "password")
361+
conf.set("spark.ssl.trustStorePassword", "password")
362+
363+
val filtered = conf.getAll.filter { case (k, _) => SparkConf.isExecutorStartupConf(k) }
364+
assert(filtered.isEmpty)
365+
}
366+
356367
test("SPARK-27244 toDebugString redacts sensitive information") {
357368
val conf = new SparkConf(loadDefaults = false)
358369
.set("dummy.password", "dummy-password")

0 commit comments

Comments
 (0)