Spark Worker not starting

[Sharathmk99]

Hi,
Spark worker is not starting. In the Spark master web ui there is no spark worker listed.
When i saw logs there was incorrect spark master url error.

Thanks

[caioquirino]

Hi! Can you attach the logs please?

[Sharathmk99]

15/12/23 10:03:18 INFO worker.Worker: Registered signal handlers for [TERM, HUP, INT]
15/12/23 10:03:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/12/23 10:03:21 INFO spark.SecurityManager: Changing view acls to: spark
15/12/23 10:03:21 INFO spark.SecurityManager: Changing modify acls to: spark
15/12/23 10:03:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
15/12/23 10:03:22 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/12/23 10:03:22 INFO Remoting: Starting remoting
15/12/23 10:03:22 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@edc0380e49b1:36922]
15/12/23 10:03:22 INFO util.Utils: Successfully started service 'sparkWorker' on port 36922.
Exception in thread "main" org.apache.spark.SparkException: Invalid master URL: spark://:
at org.apache.spark.util.Utils$.extractHostPortFromSparkUrl(Utils.scala:1981)
at org.apache.spark.deploy.master.Master$.toAkkaUrl(Master.scala:879)
at org.apache.spark.deploy.worker.Worker$$anonfun$12.apply(Worker.scala:551)
at org.apache.spark.deploy.worker.Worker$$anonfun$12.apply(Worker.scala:551)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at org.apache.spark.deploy.worker.Worker$.startSystemAndActor(Worker.scala:551)
at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:529)
at org.apache.spark.deploy.worker.Worker.main(Worker.scala)

[Sharathmk99]

I fixed the issue by adding below line to /etc/init.d/spark-worker
. /etc/profile.d/spark-env.sh

[harshagn]

Although this is a very old post, here is an answer which worked for me. Usually, that is the issue with the SPARK_MASTER_HOST, set to the hostname of the node you are using in $SPARK_HOME/conf/spark-env.sh.