[ZEPPELIN-6458] Modernize Spark cluster-manager Dockerfiles

[move-hoon]

What is this PR for?

Modernizes the Spark standalone and YARN cluster-manager Docker examples from CentOS 7 and Java 7 to the official Apache Spark 3.5.8, Scala 2.12, Java 11 Ubuntu image.

For the Hadoop 3.3.6 YARN image, this also updates the HDFS endpoint, root daemon environment, Spark worker command, and NameNode web-port mapping required for the modern runtime. The image selects the matching Hadoop archive for amd64 or arm64, validates its SHA-512 checksum, and avoids re-uploading Spark JARs to HDFS after a container restart.

The deployment guide now documents the supported versions and the Spark 3 YARN settings: spark.master=yarn with spark.submit.deployMode=client.

What type of PR is it?

Improvement

Todos

• Replace CentOS 7 and Java 7 in the standalone and YARN Docker examples
• Document Spark/Hadoop versions and Spark 3 YARN configuration
• Verify standalone and YARN Docker build and runtime smoke tests

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-6458

How should this be tested?

The following checks passed successfully:

docker build -t spark_standalone scripts/docker/spark-cluster-managers/spark_standalone
docker build -t spark_yarn scripts/docker/spark-cluster-managers/spark_yarn_cluster

Runtime smoke tests verified Spark master and worker processes plus both Spark web UIs for standalone and YARN. The YARN smoke test also verified the HDFS Spark-JAR marker and directory, and one running YARN NodeManager.

git diff --check
bash -n scripts/docker/spark-cluster-managers/spark_standalone/entrypoint.sh \
  scripts/docker/spark-cluster-managers/spark_yarn_cluster/entrypoint.sh
xmllint --noout scripts/docker/spark-cluster-managers/spark_yarn_cluster/hdfs_conf/core-site.xml
./mvnw validate -q -Dmaven.gitcommitid.skip=true

Screenshots (if appropriate)

Not applicable. This PR changes Docker development examples and documentation only.

Questions

• Do the license files need to be updated? No.
• Are there breaking changes for older versions? No. The documented host NameNode URL remains port 50070; Docker maps it to Hadoop 3's container port 9870.
• Does this need documentation? Yes. The deployment guide is updated in this PR.

[voidmatcha]

I noticed two cases in the new jar-cache handling. If the marker lookup fails, the script treats it as a missing marker and deletes and re-uploads the existing jars. If the marker remains but some jars are missing, it skips the upload.

I put together a small follow-up commit on top of this PR's head (27d4d5e) to preserve the cache on lookup errors and compare jar filenames before reuse. Feel free to use it or take any parts you find useful.

I checked these cases by running the extracted cache-handling section against a real Hadoop instance, with an injected lookup failure or a deliberately removed jar. The 10 regression tests also pass. I have not rebuilt the full image or run a Spark application with this patch.