Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

v2.2.0-kubernetes-0.4.0

Pre-release
Pre-release
Compare
Choose a tag to compare
@liyinan926 liyinan926 released this 25 Sep 22:40
· 22 commits to branch-2.2-kubernetes since this release

Major features and fixes in this release include:

  • Support for HDFS locality.
  • Added an option to use a secret to mount small files in driver and executors.
  • Support for custom Kubernetes service account for the driver pod.
  • Support for executor java options.
  • Fixed conversion from MB to MiB in driver and executor memory specification.
  • Added configuration properties for injecting arbitrary Kubernetes secrets into the driver and executors.
  • Use a headless service to give a hostname to the driver (requiring Kubernetes DNS in the cluster).
  • Improved docker image build/push flow.
  • Added the ability to fail submission if submitter-local files are provided without the resource staging server URI.
  • Added reference YAML files for RBAC configs for driver and shuffle service.

Deprecations and removals:

  • Removed support for spark.kubernetes.driver.labels, spark.kubernetes.driver.annotations, spark.kubernetes.executor.labels,
    spark.kubernetes.executor.annotations which were deprecated. They have been superseded by new properties.
    See documentation for details.