using resource staging server - driver unable to load config map or secret

[luck02]

I get this result when trying to initialize the driver:

MountVolume.SetUp failed for volume "spark-init-properties" : configmaps "kmderived-1506142014799-init-config" not found

"volumes": [
      {
        "name": "spark-init-properties",
        "configMap": {
          "name": "kmderived-1506142014799-init-config",
          "items": [
            {
              "key": "download-submitted-files",
              "path": "spark-init.properties"
            }
          ],
          "defaultMode": 420
        }

⇒  kubectl describe configmap kmderived-1506142014799-init-config
Name:		kmderived-1506142014799-init-config
Namespace:	default
Labels:		<none>
Annotations:	<none>

Data
====
download-submitted-files:
----
#Java properties built from Kubernetes config map with name: kmderived-1506142014799-init-config and config map key: $configMapKey
#Fri Sep 22 21:48:39 PDT 2017
spark.kubernetes.initcontainer.downloadJarsSecretLocation=/mnt/secrets/spark-init/downloadSubmittedJarsSecret
spark.kubernetes.initcontainer.downloadFilesSecretLocation=/mnt/secrets/spark-init/downloadSubmittedFilesSecret
spark.kubernetes.initcontainer.downloadJarsResourceIdentifier=eed35aaa-427e-478d-bef3-5a1db65027fb
spark.kubernetes.initcontainer.downloadFilesResourceIdentifier=ca549af3-f58a-4a48-bd79-749cb959c5ea
spark.ssl.kubernetes.resourceStagingServer.enabled=false
spark.kubernetes.resourceStagingServer.uri=http\://asfdsafsdaf-23423423432.us-east-1.elb.amazonaws.com\:10000
spark.kubernetes.mountdependencies.filesDownloadDir=/var/spark-data/spark-files
spark.kubernetes.mountdependencies.jarsDownloadDir=/var/spark-data/spark-jars

Events:	<none>

This config map does exist, same situation with the secret.

Also, the spark submit call hangs for quite awhile:

2017-09-22 21:46:55 WARN  ConfigurationUtils:66 - Configuration with key spark.kubernetes.driver.labels is deprecated. Use configurations with prefix spark.kubernetes.driver.label.<key> instead.
2017-09-22 21:46:55 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-09-22 21:46:55 INFO  SecurityManager:54 - Changing view acls to: garylucas
2017-09-22 21:46:55 INFO  SecurityManager:54 - Changing modify acls to: garylucas
2017-09-22 21:46:55 INFO  SecurityManager:54 - Changing view acls groups to:
2017-09-22 21:46:55 INFO  SecurityManager:54 - Changing modify acls groups to:
2017-09-22 21:46:55 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(garylucas); groups with view permissions: Set(); users  with modify permissions: Set(garylucas); groups with modify permissions: Set()
---- Output hangs here for at least 2-3 minutes
2017-09-22 21:48:41 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state:

Note the nearly 2 minutes between the last set and the pod status updates? Is that normal?

I've got about 13mb of jar's that are uploading, so that may be normal... However it seems pretty slow

      --packages org.apache.hadoop:hadoop-aws:2.7.3,joda-time:joda-time:2.9.3,mysql:mysql-connector-java:5.1.35,com.amazonaws:aws-java-sdk:1.7.4\

If it's that slow, that's fine. But I'm more wondering if something isn't timing out on the back end.

[luck02]

Is it possible it's a race condition where the pod gets created before the configmap and secret?

The pod is created @ "creationTimestamp": "2017-09-23T05:02:36Z"
The config map is created @ "creationTimestamp": "2017-09-23T05:02:37Z"
The secret is created @ "creationTimestamp": "2017-09-23T05:02:36Z"

I'm not sure what the initialization looks like for this, IE will the pod immediately error out? Will it try again etc.

[luck02]

I'm just digging around in the spark submit code, trying to figure out what the interplay between the staging server and the pod scheduling is. Looks like everyone is busy trying to merge upstream (Which is awesome!)

[luck02]

hrm. I tweaked my setup a bit.

I baked my jars into the images (which makes more sense anyways). Submission is now very fast, which is great but still the same error.

pod can't mount configmap or secret.

  ./infra/spark-2.2.0-k8s-0.4.0-SNAPSHOT-bin-2.7.3/bin/spark-submit \
      --verbose \
      --deploy-mode cluster \
      --master k8s://http://127.0.0.1:8001 \
      --kubernetes-namespace default \
      --jars local:/opt/jars/hadoop-aws-2.7.3.jar,local:/opt/jars/aws-java-sdk-1.7.4.jar,local:/opt/jars/joda-time-2.9.3.jar,local:/opt/jars/mysql-connector-java-5.1.35.jar\
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-k8s-service-account \
      --conf spark.kubernetes.docker.image.pullPolicy=Always \
      --conf spark.driver.memory=14G \
      --conf spark.driver.cores=8 \
      --conf spark.executor.instances=10 \
      --conf spark.executor.cores=6\
      --conf spark.executor.memory=20G \
      --conf spark.app.name=kmderived \
      --conf spark.kubernetes.driver.docker.image=$SPARK_DRIVER \
      --conf spark.kubernetes.executor.docker.image=$SPARK_EXECUTOR \
      --conf spark.kubernetes.initcontainer.docker.image=$SPARK_INIT \
      --conf spark.kubernetes.driver.labels=role=kissmetrics \
      --conf spark.kubernetes.executor.labels=role=kissmetrics \
      --conf spark.kubernetes.resourceStagingServer.uri=http://someaws-identifier.aws-region-1.elb.amazonaws.com:10000 \
      --py-files ./jobs.zip \
      ./driver.py

And results are:

MountVolume.SetUp failed for volume "spark-init-properties" : configmaps "kmderived-1506310930218-init-config" not found

Manually confirmed that config map does exist. My working theory is some sort of race condition.

In addition the secrets and configmaps are not cleaned up. So their left lying around after a failure.

I got to the point where I am almost starting to actually debug the behavious but it's slow going as I'm just not familiar enough with the spark codebase, scala development in general and java development (background is primarily Golang / .NET / JS with a sprinkling of scala).

[sahilprasad]

Hey @luck02 ! Not sure if I can help too much, but in my experience, this is oftentimes the by-product of a different issue than reported. Due to the lack of consistent logging and debugging tools, a smaller error might cause this ConfigMap failure to occur, while this error might not be the reason your job submission failed at all.

Regardless, can you confirm that you are prepending local://, instead of just local, in front of the pre-baked JARs that you provide to the --jars param? I.e. --jars local://{path to jar 1}, ...

[luck02]

@sahilprasad it was local:/ I changed it to local:// and got the same general issue.

I've also now tried without the staging server by cutting out all files except what i needed for this one job in the zip. IE got it down to less than 10kb.

Similar problem but totally taking the staging service out of the picture:

spark submit:

  ./infra/spark-2.2.0-k8s-0.4.0-SNAPSHOT-bin-2.7.3/bin/spark-submit \
      --verbose \
      --deploy-mode cluster \
      --master k8s://http://127.0.0.1:8001 \
      --kubernetes-namespace default \
      --jars local://opt/jars/hadoop-aws-2.7.3.jar,local://opt/jars/aws-java-sdk-1.7.4.jar,local://opt/jars/joda-time-2.9.3.jar,local://opt/jars/mysql-connector-java-5.1.35.jar\
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-k8s-service-account \
      --conf spark.kubernetes.docker.image.pullPolicy=Always \
      --conf spark.driver.memory=14G \
      --conf spark.driver.cores=8 \
      --conf spark.executor.instances=10 \
      --conf spark.executor.cores=6\
      --conf spark.executor.memory=20G \
      --conf spark.app.name=kmderived \
      --conf spark.kubernetes.driver.docker.image=$SPARK_DRIVER \
      --conf spark.kubernetes.executor.docker.image=$SPARK_EXECUTOR \
      --conf spark.kubernetes.initcontainer.docker.image=$SPARK_INIT \
      --conf spark.kubernetes.driver.labels=role=kissmetrics \
      --conf spark.kubernetes.executor.labels=role=kissmetrics \
      --py-files file://$PATH_TO_JOB \
      file://$PATH_TO_DRIVER

pod logs:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ /sbin/tini -s -- /bin/sh -c 'SPARK_CLASSPATH="${SPARK_HOME}/jars/*" &&     env | grep SPARK_JAVA_OPT_ | sed '\''s/[^=]*=\(.*\)/\1/g'\'' > /tmp/java_opts.txt &&     readarray -t SPARK_DRIVER_JAVA_OPTS < /tmp/java_opts.txt &&     if ! [ -z ${SPARK_MOUNTED_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_MOUNTED_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_SUBMIT_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_SUBMIT_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_MOUNTED_FILES_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_DIR/." .; fi &&     if ! [ -z ${SPARK_MOUNTED_FILES_FROM_SECRET_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_FROM_SECRET_DIR/." .; fi &&     ${JAVA_HOME}/bin/java "${SPARK_DRIVER_JAVA_OPTS[@]}" -cp $SPARK_CLASSPATH -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY $SPARK_DRIVER_CLASS $PYSPARK_PRIMARY $PYSPARK_FILES $SPARK_DRIVER_ARGS'
python: can't open file '/var/spark-data/spark-files/driver.py': [Errno 2] No such file or directory
Exception in thread "main" org.apache.spark.SparkUserAppException: User application exited with 2
	at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:97)
	at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)

and this is in the events:

MountVolume.SetUp failed for volume "submitted-files" : secrets "spark-9ed113390ce643e793baf9f9e83514cb-submitted-files" not found

That secret does exist, not sure where to go from here.

It's obvious the issue isn't in the staging server. I also just confirmed that the secret in question is correct. I see my driver.py source code and a blob of zipped data in the other one.

[sahilprasad]

Can you try to submit again, with one more / in each of your JARs? It should be local:// (the adapter) with the actual path appended to it. For example, local:// + /opt/spark/jars/something.jar. Just want to rule this out as a possible cause for the error.

[luck02]

Absolutely, thanks for helping @sahilprasad

I added the additional / as suggested, no change. I flat out removed the --jars line from my spark-submit script... no change.

[luck02]

Here's my hole spark-submit output, not sure if it'll help.

garylucas@gary-lucas:~/dev/ub-data-derpmetrics|DAT-spark-on-k8s⚡
⇒  ./run_page_spark.bash
/Users/garylucas/dev/ub-data-derpmetrics/jobs.zip
/Users/garylucas/dev/ub-data-derpmetrics/driver.py
docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest
docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-executor-py-derpmetrics:latest
Using properties file: null
Parsed arguments:
  master                  k8s://http://127.0.0.1:8001
  deployMode              cluster
  executorMemory          20G
  executorCores           6
  totalExecutorCores      null
  propertiesFile          null
  driverMemory            14G
  driverCores             8
  driverExtraClassPath    null
  driverExtraLibraryPath  null
  driverExtraJavaOptions  null
  supervise               false
  queue                   null
  numExecutors            10
  files                   null
  pyFiles                 file:///Users/garylucas/dev/ub-data-derpmetrics/jobs.zip
  archives                null
  mainClass               null
  primaryResource         file:///Users/garylucas/dev/ub-data-derpmetrics/driver.py
  name                    kmderived
  childArgs               []
  jars                    null
  packages                null
  packagesExclusions      null
  repositories            null
  verbose                 true

Spark properties used, including those specified through
 --conf and those from the properties file null:
  (spark.kubernetes.executor.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-executor-py-derpmetrics:latest)
  (spark.kubernetes.docker.image.pullPolicy,Always)
  (spark.executor.memory,20G)
  (spark.driver.memory,14G)
  (spark.executor.instances,10)
  (spark.kubernetes.authenticate.driver.serviceAccountName,spark-k8s-service-account)
  (spark.driver.cores,8)
  (spark.app.name,kmderived)
  (spark.kubernetes.driver.labels,role=derpmetrics)
  (spark.kubernetes.driver.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest)
  (spark.kubernetes.initcontainer.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-init-derpmetrics:latest)
  (spark.kubernetes.executor.labels,role=derpmetrics)
  (spark.executor.cores,6)


Main class:
org.apache.spark.deploy.kubernetes.submit.Client
Arguments:
--primary-py-file
file:///Users/garylucas/dev/ub-data-derpmetrics/driver.py
--main-class
org.apache.spark.deploy.PythonRunner
--other-py-files
file:///Users/garylucas/dev/ub-data-derpmetrics/jobs.zip
System properties:
(spark.kubernetes.executor.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-executor-py-derpmetrics:latest)
(spark.kubernetes.docker.image.pullPolicy,Always)
(spark.executor.memory,20G)
(spark.driver.memory,14G)
(spark.executor.instances,10)
(SPARK_SUBMIT,true)
(spark.kubernetes.namespace,default)
(spark.kubernetes.authenticate.driver.serviceAccountName,spark-k8s-service-account)
(spark.driver.cores,8)
(spark.app.name,kmderived)
(spark.kubernetes.driver.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest)
(spark.kubernetes.driver.labels,role=derpmetrics)
(spark.kubernetes.initcontainer.docker.image,docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-init-derpmetrics:latest)
(spark.submit.deployMode,cluster)
(spark.master,k8s://http://127.0.0.1:8001)
(spark.kubernetes.executor.labels,role=derpmetrics)
(spark.executor.cores,6)
Classpath elements:



2017-09-24 21:49:36 WARN  ConfigurationUtils:66 - Configuration with key spark.kubernetes.driver.labels is deprecated. Use configurations with prefix spark.kubernetes.driver.label.<key> instead.
2017-09-24 21:49:38 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state:
	 pod name: kmderived-1506314976209-driver
	 namespace: default
	 labels: role -> derpmetrics, spark-app-selector -> spark-5f563affbb7547fa9562c78bfd66c763, spark-role -> driver
	 pod uid: ef5e9e46-a1ac-11e7-b97c-026e1e112030
	 creation time: 2017-09-25T04:49:38Z
	 service account name: spark-k8s-service-account
	 volumes: submitted-files, spark-k8s-service-account-token-fn4qg
	 node name: N/A
	 start time: N/A
	 container images: N/A
	 phase: Pending
	 status: []
2017-09-24 21:49:38 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state:
	 pod name: kmderived-1506314976209-driver
	 namespace: default
	 labels: role -> derpmetrics, spark-app-selector -> spark-5f563affbb7547fa9562c78bfd66c763, spark-role -> driver
	 pod uid: ef5e9e46-a1ac-11e7-b97c-026e1e112030
	 creation time: 2017-09-25T04:49:38Z
	 service account name: spark-k8s-service-account
	 volumes: submitted-files, spark-k8s-service-account-token-fn4qg
	 node name: ip-10-253-50-77.ec2.internal
	 start time: N/A
	 container images: N/A
	 phase: Pending
	 status: []
2017-09-24 21:49:38 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state:
	 pod name: kmderived-1506314976209-driver
	 namespace: default
	 labels: role -> derpmetrics, spark-app-selector -> spark-5f563affbb7547fa9562c78bfd66c763, spark-role -> driver
	 pod uid: ef5e9e46-a1ac-11e7-b97c-026e1e112030
	 creation time: 2017-09-25T04:49:38Z
	 service account name: spark-k8s-service-account
	 volumes: submitted-files, spark-k8s-service-account-token-fn4qg
	 node name: ip-10-253-50-77.ec2.internal
	 start time: 2017-09-25T04:49:38Z
	 container images: docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest
	 phase: Pending
	 status: [ContainerStatus(containerID=null, image=docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2017-09-24 21:49:38 INFO  Client:54 - Waiting for application kmderived to finish...
2017-09-24 21:49:40 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state:
	 pod name: kmderived-1506314976209-driver
	 namespace: default
	 labels: role -> derpmetrics, spark-app-selector -> spark-5f563affbb7547fa9562c78bfd66c763, spark-role -> driver
	 pod uid: ef5e9e46-a1ac-11e7-b97c-026e1e112030
	 creation time: 2017-09-25T04:49:38Z
	 service account name: spark-k8s-service-account
	 volumes: submitted-files, spark-k8s-service-account-token-fn4qg
	 node name: ip-10-253-50-77.ec2.internal
	 start time: 2017-09-25T04:49:38Z
	 container images: docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest
	 phase: Failed
	 status: [ContainerStatus(containerID=docker://075a6b4aff0db13aa24a5acc30392227192aa7c5fd8a050156e490809fdfb930, image=docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest, imageID=docker-pullable://docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics@sha256:ec68cf8707428bf83f2fb4e0683aa94bfd7acfd0927de29212a3ee9eb3b8ecdb, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=ContainerStateTerminated(containerID=docker://075a6b4aff0db13aa24a5acc30392227192aa7c5fd8a050156e490809fdfb930, exitCode=1, finishedAt=2017-09-25T04:49:39Z, message=null, reason=Error, signal=null, startedAt=2017-09-25T04:49:39Z, additionalProperties={}), waiting=null, additionalProperties={}), additionalProperties={})]
2017-09-24 21:49:40 INFO  LoggingPodStatusWatcherImpl:54 - Container final statuses:


	 Container name: spark-kubernetes-driver
	 Container image: docker_ecr.dkr.ecr.us-east-1.amazonaws.com/spark-driver-py-derpmetrics:latest
	 Container state: Terminated
	 Exit code: 1
2017-09-24 21:49:40 INFO  Client:54 - Application kmderived finished.

I'm using a podpreset to provide the actual secrets.

Huh, I'm going to make that fail and see if anything changes (I doubt it) (Nope, had no effect).

[sahilprasad]

Of course!

Let's try keeping the --jars out of the equation, and adding the resource staging server specs back in. Not sure if the pod presets are contributing to this error, but as I mentioned, the configmap mounting is a rather cryptic error, so we can't be certain about the cause yet.

[luck02]

That gave me the following error:

MountVolume.SetUp failed for volume "spark-init-properties" : configmaps "kmderived-1506316079615-init-config" not found

I didn't see an attempt to load an actual secret, so that's different. But presumably that's just taking the --jars out of the picture.

I did however also scrape a previously failed pod creation yaml and was able to launch it. It did mount the secret / config maps. But then crashed with the following error which might be suspicious:

++ id -u
+ myuid=0
++ id -g
+ mygid=0
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ /sbin/tini -s -- /bin/sh -c 'SPARK_CLASSPATH="${SPARK_HOME}/jars/*" &&     env | grep SPARK_JAVA_OPT_ | sed '\''s/[^=]*=\(.*\)/\1/g'\'' > /tmp/java_opts.txt &&     readarray -t SPARK_DRIVER_JAVA_OPTS < /tmp/java_opts.txt &&     if ! [ -z ${SPARK_MOUNTED_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_MOUNTED_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_SUBMIT_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_SUBMIT_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_EXTRA_CLASSPATH+x} ]; then SPARK_CLASSPATH="$SPARK_EXTRA_CLASSPATH:$SPARK_CLASSPATH"; fi &&     if ! [ -z ${SPARK_MOUNTED_FILES_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_DIR/." .; fi &&     if ! [ -z ${SPARK_MOUNTED_FILES_FROM_SECRET_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_FROM_SECRET_DIR/." .; fi &&     ${JAVA_HOME}/bin/java "${SPARK_DRIVER_JAVA_OPTS[@]}" -cp $SPARK_CLASSPATH -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY $SPARK_DRIVER_CLASS $PYSPARK_PRIMARY $PYSPARK_FILES $SPARK_DRIVER_ARGS'
Error: Could not find or load main class .opt.spark.jars.RoaringBitmap-0.5.11.jar

But presumably is just related to having no startup arguments, however the config-map was in fact present.

Thanks for all the assistance, needing to sign off for the evening (PDT timezone)

[sahilprasad]

Interesting. I wonder if this is related to my PR #464 . Try adding just the spark-examples JAR with the --jars param to the spec of your latest submission, and if it works, then it is related. If not, hopefully the resulting error is more revealing of the actual issue.

[luck02]

not much in the way of meaningfull change:

#!/usr/bin/env bash

ECR_URL="514588561872.dkr.ecr.us-east-1.amazonaws.com"
SPARK_DRIVER="$ECR_URL/spark-driver-py-derpmetrics:latest"
SPARK_EXECUTOR="$ECR_URL/spark-executor-py-derpmetrics:latest"
SPARK_INIT="$ECR_URL/spark-init-derpmetrics:latest"
PATH_TO_JOB="$PWD/jobs.zip"
PATH_TO_DRIVER="$PWD/jobs/derpmetrics_warehouse/Derived_NonAppPageViewed/Compute__Derived_NonAppPageViewed.py"
PATH_TO_SPARK_EXAMPLES="$PWD/infra/spark-2.2.0-k8s-0.4.0-SNAPSHOT-bin-2.7.3/examples/jars/spark-examples_2.11-2.2.0-k8s-0.4.0-SNAPSHOT.jar"


echo $PATH_TO_JOB
echo $PATH_TO_DRIVER

echo $SPARK_DRIVER
echo $SPARK_EXECUTOR


  ./infra/spark-2.2.0-k8s-0.4.0-SNAPSHOT-bin-2.7.3/bin/spark-submit \
      --verbose \
      --deploy-mode cluster \
      --master k8s://http://127.0.0.1:8001 \
      --kubernetes-namespace default \
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-k8s-service-account \
      --conf spark.kubernetes.docker.image.pullPolicy=Always \
      --conf spark.driver.memory=4G \
      --conf spark.driver.cores=2 \
      --conf spark.executor.instances=1 \
      --conf spark.executor.cores=2\
      --conf spark.executor.memory=6G \
      --conf spark.app.name=kmderived \
      --conf spark.kubernetes.driver.docker.image=$SPARK_DRIVER \
      --conf spark.kubernetes.executor.docker.image=$SPARK_EXECUTOR \
      --conf spark.kubernetes.initcontainer.docker.image=$SPARK_INIT \
      --conf spark.kubernetes.driver.labels=role=derpmetrics \
      --conf spark.kubernetes.executor.labels=role=derpmetrics \
      --conf spark.kubernetes.resourceStagingServer.uri=http://herpip-421161154.us-east-1.elb.amazonaws.com:10000 \
      --jars file://$PATH_TO_SPARK_EXAMPLES\
      file://$PATH_TO_DRIVER

Results in:
MountVolume.SetUp failed for volume "spark-init-properties" : configmaps "kmderived-1506351376470-init-config" not found

And pod failed to initialize.

Just reading your PR now

[luck02]

Your PR doesn't look overly related but who knows, I tried one more time with just my python file and still got the same error.

MountVolume.SetUp failed for volume "spark-init-properties" : configmaps "kmderived-1506353114000-init-config" not found

I'm still wondering about mount order etc. Correct me if I'm wrong but the problem needs to be happening in the context of 'spark-submit'

I got to the point of being able to test out a spark submit with spark running locally in debug mode, but hadn't gotten to the point of doing any significant debugging. Given that my submit steps are so simple I'll probably wire up my spark submit args to local-dev and start trying to troubleshoot there. It seems as if the secret / config map is being created after the pod spec is executed. But presumably that's just behaviour, not root cause. Will update if I find anything and if we get another spark 2.2 cut I'll try that too. I did run integration tests last night against my local build and all was 'good'... Also, when I bake all my code into the images it seems to work with the exception of: #504

In fact, that blockage was why I tried to move to submitting local resources. If I wanted to debug that issue I really wanted to be able to iterate faster.

I'm wondering if perhaps my local snapshot build is broken in some subtle way:

garylucas@gary-lucas:~/dev/spark|branch-2.2-kubernetes⚡
⇒  git rev-parse HEAD
8a0f485d83f84a9b556f7634fed8cb1b65e12159
garylucas@gary-lucas:~/dev/spark|branch-2.2-kubernetes⚡
⇒  ./dev/make-distribution.sh --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3

As mentioned in #504 I'm building my own due to broken service account issues in the release. If a new build gets cut I'll try that.

[mccheah]

The pod does get created before we create the config map and secret. This is because we need the Pod's UID to create the owner reference graph from the config map and secret back up to the pod. But if the pod is created first, it should eventually recover after the API server detects that the dependent objects are created even after the fact. @foxish - could this perhaps be a bug in Kubernetes? @luck02 what Kubernetes version is being used?