|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: "Keeping Kubernetes Secure with Updated Go Versions" |
| 4 | +date: 2023-04-06 |
| 5 | +slug: keeping-kubernetes-secure-with-updated-go-versions |
| 6 | +--- |
| 7 | + |
| 8 | +**Author**: [Jordan Liggitt](https://github.com/liggitt) (Google) |
| 9 | + |
| 10 | +### The problem |
| 11 | + |
| 12 | +Since v1.19 (released in 2020), the Kubernetes project provides 12-14 months of patch releases for each minor version. |
| 13 | +This enables users to qualify and adopt Kubernetes versions in an annual upgrade cycle and receive security fixes for a year. |
| 14 | + |
| 15 | +The [Go project](https://github.com/golang/go/wiki/Go-Release-Cycle#release-maintenance) releases new minor versions twice a year, |
| 16 | +and provides security fixes for the last two minor versions, resulting in about a year of support for each Go version. |
| 17 | +Even though each new Kubernetes minor version is built with a supported Go version when it is first released, |
| 18 | +that Go version falls out of support before the Kubernetes minor version does, |
| 19 | +and the lengthened Kubernetes patch support since v1.19 only widened that gap. |
| 20 | + |
| 21 | +At the time this was written, just over half of all [Go patch releases](https://go.dev/doc/devel/release) (88/171) have contained fixes for issues with possible security implications. |
| 22 | +Even though many of these issues were not relevant to Kubernetes, some were, so it remained important to use supported Go versions that received those fixes. |
| 23 | + |
| 24 | +An obvious solution would be to simply update Kubernetes release branches to new minor versions of Go. |
| 25 | +However, Kubernetes avoids [destabilizing changes in patch releases](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md#what-kind-of-prs-are-good-for-cherry-picks), |
| 26 | +and historically, this prevented updating existing release branches to new minor versions of Go, due to changes that were considered prohibitively complex, risky, or breaking to include in a patch release. |
| 27 | +Examples include: |
| 28 | + |
| 29 | +* Go 1.6: enabling http/2 by default |
| 30 | +* Go 1.14: EINTR handling issues |
| 31 | +* Go 1.17: dropping x509 CN support, ParseIP changes |
| 32 | +* Go 1.18: disabling x509 SHA-1 certificate support by default |
| 33 | +* Go 1.19: dropping current-dir LookPath behavior |
| 34 | + |
| 35 | +Some of these changes could be easily mitigated in Kubernetes code, |
| 36 | +some could only be opted out of via a user-specified `GODEBUG` envvar, |
| 37 | +and others required invasive code changes or could not be avoided at all. |
| 38 | +Because of this inconsistency, Kubernetes release branches have typically remained on a single Go minor version, |
| 39 | +and risked being unable to pick up relevant Go security fixes for the last several months of each Kubernetes minor version's support lifetime. |
| 40 | + |
| 41 | +When a relevant Go security fix was only available in newer Kubernetes minor versions, |
| 42 | +users would have to upgrade away from older Kubernetes minor versions before their 12-14 month support period ended, just to pick up those fixes. |
| 43 | +If a user was not prepared to do that upgrade, it could result in vulnerable Kubernetes clusters. |
| 44 | +Even if a user could accommodate the unexpected upgrade, the uncertainty made Kubernetes' annual support less reliable for planning. |
| 45 | + |
| 46 | +### The solution |
| 47 | + |
| 48 | +We're happy to announce that the gap between supported Kubernetes versions and supported Go versions has been resolved as of January 2023. |
| 49 | + |
| 50 | +We worked closely with the Go team over the past year to address the difficulties adopting new Go versions. |
| 51 | +This prompted a [discussion](https://github.com/golang/go/discussions/55090), [proposal](https://github.com/golang/go/issues/56986), |
| 52 | +[talk at GopherCon](https://www.youtube.com/watch?v=v24wrd3RwGo), and a [design](https://go.dev/design/56986-godebug) for improving backward compatibility in Go, |
| 53 | +ensuring new Go versions can maintain compatible runtime behavior with previous Go versions for a minimum of two years (four Go releases). |
| 54 | +This allows projects like Kubernetes to update release branches to supported Go versions without exposing users to behavior changes. |
| 55 | + |
| 56 | +The proposed improvements are on track to be [included in Go 1.21](https://tip.golang.org/doc/godebug), and the Go team already delivered targeted compatibility improvements in a Go 1.19 patch release in late 2022. |
| 57 | +Those changes enabled Kubernetes 1.23+ to update to Go 1.19 in January of 2023, while avoiding any user-facing configuration or behavior changes. |
| 58 | +All supported Kubernetes release branches now use supported Go versions, and can pick up new Go patch releases with available security fixes. |
| 59 | + |
| 60 | +Going forward, Kubernetes maintainers remain committed to making Kubernetes patch releases as safe and non-disruptive as possible, |
| 61 | +so there are several requirements a new Go minor version must meet before existing Kubernetes release branches will update to use it: |
| 62 | + |
| 63 | +1. The new Go version must be available for at least 3 months. |
| 64 | + This gives time for adoption by the Go community, and for reports of issues or regressions. |
| 65 | +2. The new Go version must be used in a new Kubernetes minor release for at least 1 month. |
| 66 | + This ensures all Kubernetes release-blocking tests pass on the new Go version, |
| 67 | + and gives time for feedback from the Kubernetes community on release candidates and early adoption of the new minor release. |
| 68 | +3. There must be no regressions from the previous Go version known to impact Kubernetes. |
| 69 | +4. Runtime behavior must be preserved by default, without requiring any action by Kubernetes users / administrators. |
| 70 | +5. Kubernetes libraries like `k8s.io/client-go` must remain compatible with the original Go version used for each minor release, |
| 71 | + so consumers won't *have* to update Go versions to pick up a library patch release (though they are encouraged to build with supported Go versions, |
| 72 | + which is made even easier with the [compatibility improvements](https://go.dev/design/56986-godebug) planned in Go 1.21). |
| 73 | + |
| 74 | +The goal of all of this work is to unobtrusively make Kubernetes patch releases safer and more secure, |
| 75 | +and to make Kubernetes minor versions safe to use for the entire duration of their support lifetime. |
| 76 | + |
| 77 | +Many thanks to the Go team, especially Russ Cox, for helping drive these improvements in ways that will benefit all Go users, not just Kubernetes. |
0 commit comments