Spark History Server support for Azure Data Lake Storage Gen2 (ADLS) #464
Description
Spark application logs should be storable in Azure Data Lake Storage Gen2 (ADLS).
The hadoop-azure module must be added to the Spark image and the spec.logFileDirectory structure could be extended with adls. Currently only s3 is supported. Alternatively, a custom log directory should be specifiable.
The SAS token should be read from a Secret.
- feat: Make spark-env.sh configurable #473
- stackabletech/decisions#32
- feat: custom log directory #479 (including an integration test for a custom log directory in HDFS)
- Test manually a custom log directory in ADLS with
spark-k8s-operator:0.0.0-pr479
SparkHistoryServer: https://github.com/stackabletech/sbernauer-customers/blob/e2e6db97645ac2d0fd54130c81e45a8d3e7107e7/airplus/poc-cluster/specs/spark-history.yaml
SparkApplication: https://github.com/stackabletech/sbernauer-customers/blob/e2e6db97645ac2d0fd54130c81e45a8d3e7107e7/airplus/poc-cluster/test/spark.yaml
Documentation
https://docs.stackable.tech/home/nightly/spark-k8s/usage-guide/history-server#_custom_log_directory
Release Notes
New / extended platform features
Other product features
- Apache Spark: A custom log directory can be specified for the event logs, allowing to choose another location than an S3 bucket, e.g. HDFS or ABFS.