Finish dynamic client mode

[GeorgeJahad]

Dynamic allocation allocates the min # of executors as part of of the initial gang, but subsequent executors are not part of the gang.

instead the use the new Armada env vars as node selectors:
$ARMADA_GANG_NODE_UNIFORMITY_LABEL_NAME: $ARMADA_GANG_NODE_UNIFORMITY_LABEL_VALUE.

in dynamic cluster mode the driver is running on the cluster and has access to those env vars, but in dynamic client mode it doesn't.

To support that we need to modify the executor registration process to pass the vars to the driver so that it can start subsequent executors with them.

[GeorgeJahad]

When using dynamic allocation, new executors need to be allocated on the same cluster as the old. In the dynamicCluster implementation, those are determined using the environment variables received by the driver like so:
https://github.com/armadaproject/armada-spark/blob/98ab4083212f5f4aabaec88b0a8347baa3c67d7c/src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaClientApplication.scala#L1296-L1303

This works because in the dynamicCluster implementation, the driver is running in the cluster as well, and so it gets those env vars naturally when the pod is initialized, just as the executors do.

But in the dynamicClient implementation, the driver is running on the host/devpod and doesn't have access to those env vars. Instead it must retrieve them from the first executor that registers itself.

Currently all executors in spark send all env vars with the "SPARK_EXECUTOR_ATTRIBUTE_" prefix to the driver when it registers with the driver:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L156-L160

So the goal here is to generate env vars like: "SPARK_EXECUTOR_ATTRIBUTE_ARMADA_GANG_NODE_UNIFORMITY_LABEL_NAME" so that the driver receives them and parses them like the env vars it receives naturally in dynamicCluster mode.

The easiest way to do this may be in the entrypoint.sh script that executors run before starting the vm.

[GeorgeJahad]

entrypoint.sh changes could look something like this:

+# Copy ARMADA_GANG* env vars to SPARK_EXECUTOR_ATTRIBUTE_* so they get sent to driver
+for var in $(env | grep '^ARMADA_GANG' | cut -d= -f1); do
+  export "SPARK_EXECUTOR_ATTRIBUTE_$var=${!var}"
+done
+
 case "$1" in
   driver)
     shift 1