v2.3.0

Highlights

• Support Spark v4.
• Add support for Spark Connect by adding a new CRD called SparkConnect. One example can be found here.
• Upgrade CRDs automatically when running helm upgrade by setting hook.upgradeCrd=true. This will create a Helm pre-install/pre-upgrade Job to run kubectl apply --server-side to update CRDs.
• Configure logging format by setting {controller,webhook}.logEncoder to json or console.

Features

• Add support for Spark Connect (#2569 by @ChenYi015)
• upgrade to Spark 4.0.0 (#2564 by @nabuskey)
• Make logging encoder configurable (#2580 by @ChenYi015)
• Include pod.Status.Message in recordExecutorEvent (#2589 by @matschaffer-roblox)
• Add print columns for Spark Connect (#2592 by @nabuskey
• chore: update prometheus pattern and labels for structured streaming driver (#2581 by @yahwang)
• Add Helm hook to upgrade CRDs (#2371 by @ChenYi015)

Bug Fixes

• Splat recordExecutorEvent args for cleaner event messages (#2582 by @matschaffer-roblox)
• fix: should add executor env when driver env is empty (#2586 by @ChenYi015)
• Add web UI configurations when enabling UI service and ingress (#2599 by @ChenYi015)
• Grant get/patch permissions for SparkConnect CRD to hook ClusterRole (#2605 by @ChenYi015)

Dependencies

• Bump aquasecurity/trivy-action from 0.31.0 to 0.32.0 (#2585 by @dependabot[bot])
• Bump github.com/go-viper/mapstructure/v2 from 2.2.1 to 2.3.0 (#2572 by @dependabot[bot])
• Bump sigs.k8s.io/yaml from 1.4.0 to 1.5.0 (#2577 by @dependabot[bot])
• Bump helm.sh/helm/v3 from 3.17.3 to 3.18.4 (#2587 by @dependabot[bot])
• Bump golang.org/x/mod from 0.25.0 to 0.26.0 (#2608 by @dependabot[bot])
• Bump github.com/onsi/gomega from 1.36.1 to 1.37.0 (#2607 by @dependabot[bot])

Misc

• Add SparkConnect e2e test (#2578 by @ChenYi015)
• feat(docs): Guide to report security vulnerabilities (#2593 by @andreyvelich)
• Read Helm version from go.mod file (#2598 by @ChenYi015)

Full Changelog