|
| 1 | +# KEP-3156: HTTP3 |
| 2 | + |
| 3 | +<!-- toc --> |
| 4 | +- [Release Signoff Checklist](#release-signoff-checklist) |
| 5 | +- [Summary](#summary) |
| 6 | +- [Motivation](#motivation) |
| 7 | + - [Goals](#goals) |
| 8 | + - [Non-Goals](#non-goals) |
| 9 | +- [Proposal](#proposal) |
| 10 | + - [User Stories (Optional)](#user-stories-optional) |
| 11 | + - [Story 1](#story-1) |
| 12 | + - [Story 2](#story-2) |
| 13 | + - [Risks and Mitigations](#risks-and-mitigations) |
| 14 | +- [Design Details](#design-details) |
| 15 | + - [kube-apiserver](#kube-apiserver) |
| 16 | + - [client-go](#client-go) |
| 17 | + - [Test Plan](#test-plan) |
| 18 | + - [Graduation Criteria](#graduation-criteria) |
| 19 | + - [Alpha](#alpha) |
| 20 | + - [Beta](#beta) |
| 21 | + - [GA](#ga) |
| 22 | + - [Deprecation](#deprecation) |
| 23 | + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) |
| 24 | + - [Version Skew Strategy](#version-skew-strategy) |
| 25 | +- [Implementation History](#implementation-history) |
| 26 | +- [Drawbacks](#drawbacks) |
| 27 | +- [Alternatives](#alternatives) |
| 28 | +<!-- /toc --> |
| 29 | + |
| 30 | +## Release Signoff Checklist |
| 31 | + |
| 32 | +Items marked with (R) are required *prior to targeting to a milestone / release*. |
| 33 | + |
| 34 | +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) |
| 35 | +- [ ] (R) KEP approvers have approved the KEP status as `implementable` |
| 36 | +- [ ] (R) Design details are appropriately documented |
| 37 | +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) |
| 38 | + - [ ] e2e Tests for all Beta API Operations (endpoints) |
| 39 | + - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) |
| 40 | + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free |
| 41 | +- [ ] (R) Graduation criteria is in place |
| 42 | + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) |
| 43 | +- [ ] (R) Production readiness review completed |
| 44 | +- [ ] (R) Production readiness review approved |
| 45 | +- [ ] "Implementation History" section is up-to-date for milestone |
| 46 | +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] |
| 47 | +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes |
| 48 | + |
| 49 | +[kubernetes.io]: https://kubernetes.io/ |
| 50 | +[kubernetes/enhancements]: https://git.k8s.io/enhancements |
| 51 | +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes |
| 52 | +[kubernetes/website]: https://git.k8s.io/website |
| 53 | + |
| 54 | +## Summary |
| 55 | + |
| 56 | +IMPORTANT: THIS KEP HAS BEEN DEFERRED UNTIL HTTP/3 IS SUPPORTED BY THE GOLANG STANDARD LIBRARY |
| 57 | + |
| 58 | +One of the main advantages of HTTP/3 is increased performance, specifically around fetching multiple |
| 59 | +objects simultaneously. With HTTP/2, any interruption (packet loss) in the TCP connection blocks all |
| 60 | +streams (Head of line blocking). |
| 61 | + |
| 62 | +In addition, HTTP/3 offers 0-RTT support, which means that subsequent connections can start up much |
| 63 | +faster by eliminating the TLS acknowledgement from the server when setting up the connection. This |
| 64 | +means the client can start requesting data much faster than with a full TLS negotiation. |
| 65 | + |
| 66 | +## Motivation |
| 67 | + |
| 68 | +Kubernetes clusters defaulted to HTTP/2 for the control plane communications to solve all the |
| 69 | +performance and reliability problems caused by the previous protocol HTTP/1. |
| 70 | +However, there are some notable exceptions: |
| 71 | + |
| 72 | +- [communication between apiserver and webhooks](https://github.com/kubernetes/kubernetes/pull/82090). |
| 73 | +- [communication that use SPDY or websockets](https://github.com/kubernetes/kubernetes/issues/7452) |
| 74 | + |
| 75 | +HTTP/3 solves some of the HTTP/2 problems, mainly the Head of Line Blocking problem because it uses |
| 76 | +QUIC RFC9000 for the transport layer, improving the performance and the resilience of the control |
| 77 | +plane. |
| 78 | + |
| 79 | +Despite the HTTP/3 protocol, as January 2022, is still an [Internet Draft](https://datatracker.ietf.org/doc/draft-ietf-quic-http/), it is already supported by 73% of running web browsers, and there is also a [golang implementation](https://github.com/lucas-clemente/quic-go) that is being used by known projects and companies. |
| 80 | + |
| 81 | +Having support for HTTP/3 will allow the project to have early feedback, and the possibility to influence it. |
| 82 | + |
| 83 | +### Goals |
| 84 | + |
| 85 | +- Support HTTP/3 on the apiserver wherever is possible |
| 86 | +- Offer an opt-in option in client-go to use HTTP/3 |
| 87 | + |
| 88 | +### Non-Goals |
| 89 | + |
| 90 | +- Use or default any of the Kubernetes components to HTTP/3 |
| 91 | +- Automatically upgrade connections to HTTP/3 |
| 92 | +- Replace websockets or SPDY communications |
| 93 | + |
| 94 | +## Proposal |
| 95 | + |
| 96 | +The proposal is two fold: |
| 97 | + |
| 98 | +- support HTTP/3 in the apiserver, that will require to listen on the UDP port, in addition to current TCP port that is being used. |
| 99 | + |
| 100 | +- support HTTP/3 in client-go, so developers can consume it, that will require to add a new roundtripper and a new configuration option to opt-in. |
| 101 | + |
| 102 | + |
| 103 | +### User Stories (Optional) |
| 104 | + |
| 105 | +#### Story 1 |
| 106 | + |
| 107 | +- As a Kubernetes admin I'd like to be able to use HTTP/3 in front of my control-plane, so external users can benefit of its properties. |
| 108 | + |
| 109 | +#### Story 2 |
| 110 | + |
| 111 | +- As a Kubernetes developer I'd like to be able to use HTTP/3 as my communication protocol in my operator, to be more resilient on environments with hostile network conditions. |
| 112 | + |
| 113 | +### Risks and Mitigations |
| 114 | + |
| 115 | +This is a new protocol and an experimental feature, it will be behind a feature flag and will not graduate to beta until |
| 116 | +the protocol and its implementation are mature. |
| 117 | + |
| 118 | +The protocol is still a Draft, however, it is not likely it change too much at this stage before is published as RFC. |
| 119 | + |
| 120 | +The golang implementation tries to be as much compatible as possible with the golang standard library, but it is not fully compatible. It also brings a considerable amount of dependencies. |
| 121 | + |
| 122 | +## Design Details |
| 123 | + |
| 124 | +HTTP/3 main difference is that it uses UDP as transport instead of TCP, beside that, it is mostly compatible with HTTP/2. |
| 125 | + |
| 126 | +The current golang http/3 implementation tries to be as much compatible as possible with the standard library, something that |
| 127 | +simplifies the implementation, a working prototype can be seen here https://github.com/kubernetes/kubernetes/pull/106707/. |
| 128 | + |
| 129 | +### kube-apiserver |
| 130 | + |
| 131 | + |
| 132 | +A new UDP listener should be added to the kube-apiserver in the same port and address that the current TCP listener. |
| 133 | + |
| 134 | +```go |
| 135 | +// https://github.com/kubernetes/kubernetes/blob/1367cca8fd67b09606b01c0a9e46cef59aef3424/staging/src/k8s.io/apiserver/pkg/server/secure_serving.go#L276 |
| 136 | +func RunServer( |
| 137 | + server *http.Server, |
| 138 | + ln net.Listener, |
| 139 | + shutDownTimeout time.Duration, |
| 140 | + stopCh <-chan struct{}, |
| 141 | +) (<-chan struct{}, <-chan struct{}, error) { |
| 142 | +``` |
| 143 | +
|
| 144 | +In addition to the listener, there is some logic inside the apiserver that discriminates by protocol like `WrapForHTTP1Or2` that should be adapted. |
| 145 | +
|
| 146 | +```go |
| 147 | +// https://github.com/kubernetes/kubernetes/blob/1367cca8fd67b09606b01c0a9e46cef59aef3424/staging/src/k8s.io/apiserver/pkg/endpoints/responsewriter/wrapper.go#L58 |
| 148 | +// WrapForHTTP1Or2 accepts a user-provided decorator of an "inner" http.responseWriter |
| 149 | +// object and potentially wraps the user-provided decorator with a new http.ResponseWriter |
| 150 | +// object that implements http.CloseNotifier, http.Flusher, and/or http.Hijacker by |
| 151 | +// delegating to the user-provided decorator (if it implements the relevant method) or |
| 152 | +// the inner http.ResponseWriter (otherwise), so that the returned http.ResponseWriter |
| 153 | +// object implements the same subset of those interfaces as the inner http.ResponseWriter. |
| 154 | +... |
| 155 | +func WrapForHTTP1Or2(decorator UserProvidedDecorator) http.ResponseWriter { |
| 156 | +``` |
| 157 | +
|
| 158 | +### client-go |
| 159 | +
|
| 160 | +The golang http/3 implementation exposes a Transport roundtripper, there are two options to use it in client-go: |
| 161 | +
|
| 162 | +1. Use the WrapTransport option and let users configure the roundtripper manually. |
| 163 | +
|
| 164 | +```go |
| 165 | + tlsConfig, err := TLSConfigFor(config) |
| 166 | + if err != nil { |
| 167 | + return nil, err |
| 168 | + } |
| 169 | + rt = &http3.RoundTripper{ |
| 170 | + TLSClientConfig: tlsConfig, |
| 171 | + QuicConfig: &quic.Config{ |
| 172 | + KeepAlive: true, |
| 173 | + }, |
| 174 | + } |
| 175 | + |
| 176 | +config.WrapTransport = rt |
| 177 | +``` |
| 178 | +
|
| 179 | +
|
| 180 | +2. Add a new option to the RESTConfig of client-go to enable http3 and automate the HTTP3 roundtripper configuration: |
| 181 | +
|
| 182 | +```go |
| 183 | +// staging/src/k8s.io/client-go/transport/config.go |
| 184 | +// Config holds various options for establishing a transport. |
| 185 | +type Config struct { |
| 186 | + // UserAgent is an optional field that specifies the caller of this |
| 187 | + // request. |
| 188 | + UserAgent string |
| 189 | + // The base TLS configuration for this transport. |
| 190 | + TLS TLSConfig |
| 191 | + |
| 192 | +// Use HTTP3 |
| 193 | + EnableHTTP3 bool |
| 194 | +``` |
| 195 | +
|
| 196 | +### Test Plan |
| 197 | +
|
| 198 | +All current tests should pass using the new protocol, this can be done by enabling by default http3 in client go so all the |
| 199 | +components of the cluster switch to it. |
| 200 | +
|
| 201 | +### Graduation Criteria |
| 202 | +
|
| 203 | +#### Alpha |
| 204 | +
|
| 205 | +- Feature implemented in both apiserver and client-go behind a feature flag |
| 206 | +- Initial e2e tests completed and enabled |
| 207 | +
|
| 208 | +#### Beta |
| 209 | +
|
| 210 | +- HTTP/3 becomes an Internet RFC |
| 211 | +- Golang community decides on HTTP/3 support in the standard library |
| 212 | +
|
| 213 | +#### GA |
| 214 | +
|
| 215 | +#### Deprecation |
| 216 | +
|
| 217 | +### Upgrade / Downgrade Strategy |
| 218 | +
|
| 219 | +HTTP Alternative Services has become the primary mechanism for HTTP/3 upgrade, but is explicitly |
| 220 | +listed as a non-goal to implement Alt-Svc and automatic connection upgrades. |
| 221 | +
|
| 222 | +### Version Skew Strategy |
| 223 | +
|
| 224 | +## Implementation History |
| 225 | +
|
| 226 | +- (2021/1/18) Proposal |
| 227 | +## Drawbacks |
| 228 | +
|
| 229 | +The highest risk is that HTTP/3 does not become an RFC, that is very unlikely since the browsers and some of the main companies are already using it. |
| 230 | +## Alternatives |
0 commit comments