|
| 1 | +--- |
| 2 | +layout: blog |
| 3 | +title: 'Kubernetes 1.31: Streaming Transitions from SPDY to WebSockets' |
| 4 | +date: 2024-08-20 |
| 5 | +slug: websockets-transition |
| 6 | +author: > |
| 7 | + [Sean Sullivan](https://github.com/seans3) (Google) |
| 8 | + [Shannon Kularathna](https://github.com/shannonxtreme) (Google) |
| 9 | +--- |
| 10 | + |
| 11 | +In Kubernetes 1.31, by default kubectl now uses the WebSocket protocol |
| 12 | +instead of SPDY for streaming. |
| 13 | + |
| 14 | +This post describes what these changes mean for you and why these streaming APIs |
| 15 | +matter. |
| 16 | + |
| 17 | +## Streaming APIs in Kubernetes |
| 18 | + |
| 19 | +In Kubernetes, specific endpoints that are exposed as an HTTP or RESTful |
| 20 | +interface are upgraded to streaming connections, which require a streaming |
| 21 | +protocol. Unlike HTTP, which is a request-response protocol, a streaming |
| 22 | +protocol provides a persistent connection that's bi-directional, low-latency, |
| 23 | +and lets you interact in real-time. Streaming protocols support reading and |
| 24 | +writing data between your client and the server, in both directions, over the |
| 25 | +same connection. This type of connection is useful, for example, when you create |
| 26 | +a shell in a running container from your local workstation and run commands in |
| 27 | +the container. |
| 28 | + |
| 29 | +## Why change the streaming protocol? |
| 30 | + |
| 31 | +Before the v1.31 release, Kubernetes used the SPDY/3.1 protocol by default when |
| 32 | +upgrading streaming connections. SPDY/3.1 has been deprecated for eight years, |
| 33 | +and it was never standardized. Many modern proxies, gateways, and load balancers |
| 34 | +no longer support the protocol. As a result, you might notice that commands like |
| 35 | +`kubectl cp`, `kubectl attach`, `kubectl exec`, and `kubectl port-forward` |
| 36 | +stop working when you try to access your cluster through a proxy or gateway. |
| 37 | + |
| 38 | +As of Kubernetes v1.31, SIG API Machinery has modified the streaming |
| 39 | +protocol that a Kubernetes client (such as `kubectl`) uses for these commands |
| 40 | +to the more modern [WebSocket streaming protocol](https://datatracker.ietf.org/doc/html/rfc6455). |
| 41 | +The WebSocket protocol is a currently supported standardized streaming protocol |
| 42 | +that guarantees compatibility and interoperability with different components and |
| 43 | +programming languages. The WebSocket protocol is more widely supported by modern |
| 44 | +proxies and gateways than SPDY. |
| 45 | + |
| 46 | +## How streaming APIs work |
| 47 | + |
| 48 | +Kubernetes upgrades HTTP connections to streaming connections by adding |
| 49 | +specific upgrade headers to the originating HTTP request. For example, an HTTP |
| 50 | +upgrade request for running the `date` command on an `nginx` container within |
| 51 | +a cluster is similar to the following: |
| 52 | + |
| 53 | +```console |
| 54 | +$ kubectl exec -v=8 nginx -- date |
| 55 | +GET https://127.0.0.1:43251/api/v1/namespaces/default/pods/nginx/exec?command=date… |
| 56 | +Request Headers: |
| 57 | + Connection: Upgrade |
| 58 | + Upgrade: websocket |
| 59 | + Sec-Websocket-Protocol: v5.channel.k8s.io |
| 60 | + User-Agent: kubectl/v1.31.0 (linux/amd64) kubernetes/6911225 |
| 61 | +``` |
| 62 | + |
| 63 | +If the container runtime supports the WebSocket streaming protocol and at least |
| 64 | +one of the subprotocol versions (e.g. `v5.channel.k8s.io`), the server responds |
| 65 | +with a successful `101 Switching Protocols` status, along with the negotiated |
| 66 | +subprotocol version: |
| 67 | + |
| 68 | +```console |
| 69 | +Response Status: 101 Switching Protocols in 3 milliseconds |
| 70 | +Response Headers: |
| 71 | + Upgrade: websocket |
| 72 | + Connection: Upgrade |
| 73 | + Sec-Websocket-Accept: j0/jHW9RpaUoGsUAv97EcKw8jFM= |
| 74 | + Sec-Websocket-Protocol: v5.channel.k8s.io |
| 75 | +``` |
| 76 | + |
| 77 | +At this point the TCP connection used for the HTTP protocol has changed to a |
| 78 | +streaming connection. Subsequent STDIN, STDOUT, and STDERR data (as well as |
| 79 | +terminal resizing data and process exit code data) for this shell interaction is |
| 80 | +then streamed over this upgraded connection. |
| 81 | + |
| 82 | +## How to use the new WebSocket streaming protocol |
| 83 | + |
| 84 | +If your cluster and kubectl are on version 1.29 or later, there are two |
| 85 | +control plane feature gates and two kubectl environment variables that |
| 86 | +govern the use of the WebSockets rather than SPDY. In Kubernetes 1.31, |
| 87 | +all of the following feature gates are in beta and are enabled by |
| 88 | +default: |
| 89 | + |
| 90 | +- [Feature gates](/docs/reference/command-line-tools-reference/feature-gates/) |
| 91 | + - `TranslateStreamCloseWebsocketRequests` |
| 92 | + - `.../exec` |
| 93 | + - `.../attach` |
| 94 | + - `PortForwardWebsockets` |
| 95 | + - `.../port-forward` |
| 96 | +- kubectl feature control environment variables |
| 97 | + - `KUBECTL_REMOTE_COMMAND_WEBSOCKETS` |
| 98 | + - `kubectl exec` |
| 99 | + - `kubectl cp` |
| 100 | + - `kubectl attach` |
| 101 | + - `KUBECTL_PORT_FORWARD_WEBSOCKETS` |
| 102 | + - `kubectl port-forward` |
| 103 | + |
| 104 | +If you're connecting to an older cluster but can manage the feature gate |
| 105 | +settings, turn on both `TranslateStreamCloseWebsocketRequests` (added in |
| 106 | +Kubernetes v1.29) and `PortForwardWebsockets` (added in Kubernetes |
| 107 | +v1.30) to try this new behavior. Version 1.31 of `kubectl` can automatically use |
| 108 | +the new behavior, but you do need to connect to a cluster where the server-side |
| 109 | +features are explicitly enabled. |
| 110 | + |
| 111 | +## Learn more about streaming APIs |
| 112 | + |
| 113 | +- [KEP 4006 - Transitioning from SPDY to WebSockets](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets) |
| 114 | +- [RFC 6455 - The WebSockets Protocol](https://datatracker.ietf.org/doc/html/rfc6455) |
| 115 | +- [Container Runtime Interface streaming explained](https://kubernetes.io/blog/2024/05/01/cri-streaming-explained/) |
0 commit comments