|
| 1 | +--- |
| 2 | +reviewers: |
| 3 | +- jpbetz |
| 4 | +title: Mixed Version Proxy |
| 5 | +content_type: concept |
| 6 | +weight: 220 |
| 7 | +--- |
| 8 | + |
| 9 | +<!-- overview --> |
| 10 | +{{< feature-state state="alpha" for_k8s_version="v1.28" >}} |
| 11 | + |
| 12 | +Kubernetes {{< skew currentVersion >}} includes an alpha feature that lets a |
| 13 | +{{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}} |
| 14 | +proxy a resource requests to other _peer_ API servers. This is useful when there are multiple |
| 15 | +API servers running different versions of Kubernetes in one cluster (for example, during a long-lived |
| 16 | +rollout to a new release of Kubernetes). |
| 17 | + |
| 18 | +This enables cluster administrators to configure highly available clusters that can be upgraded |
| 19 | +more safely, by directing resource requests (made during the upgrade) to the correct kube-apiserver. |
| 20 | +That proxying prevents users from seeing unexpected 404 Not Found errors that stem |
| 21 | +from the upgrade process. |
| 22 | + |
| 23 | +This mechanism is called the _Mixed Version Proxy_. |
| 24 | + |
| 25 | +## Enabling the Mixed Version Proxy |
| 26 | +Ensure that `UnknownVersionInteroperabilityProxy` [feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) |
| 27 | +is enabled when you start the {{< glossary_tooltip text="API Server" term_id="kube-apiserver" >}}: |
| 28 | + |
| 29 | +```shell |
| 30 | +kube-apiserver \ |
| 31 | +--feature-gates=UnknownVersionInteroperabilityProxy=true \ |
| 32 | +# required command line arguments for this feature |
| 33 | +--peer-ca-file=<path to kube-apiserver CA cert> |
| 34 | +--proxy-client-cert-file=<path to aggregator proxy cert>, |
| 35 | +--proxy-client-key-file=<path to aggregator proxy key>, |
| 36 | +--requestheader-client-ca-file=<path to aggregator CA cert>, |
| 37 | +# requestheader-allowed-names can be set to blank to allow any Common Name |
| 38 | +--requestheader-allowed-names=<valid Common Names to verify proxy client cert against>, |
| 39 | + |
| 40 | +# optional flags for this feature |
| 41 | +--peer-advertise-ip=`IP of this kube-apiserver that should be used by peers to proxy requests` |
| 42 | +--peer-advertise-port=`port of this kube-apiserver that should be used by peers to proxy requests` |
| 43 | + |
| 44 | +# …and other flags as usual |
| 45 | +``` |
| 46 | + |
| 47 | +### Proxy transport and authentication between API servers {#transport-and-authn} |
| 48 | + |
| 49 | +* The source kube-apiserver reuses the [existing APIserver client authentication flags](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#kubernetes-apiserver-client-authentication) `--proxy-client-cert-file` and `--proxy-client-key-file` to present its identity that will be verified by its peer (the destination kube-apiserver). The destination API server verifies that peer connection based on the configuration you specify using the `--requestheader-client-ca-file` command line argument. |
| 50 | + |
| 51 | +* To authenticate the destination server's serving certs, you must configure a certificate authority bundle by specifying the `--peer-ca-file` command line argument to the **source** API server. |
| 52 | + |
| 53 | +### Configuration for peer API server connectivity |
| 54 | + |
| 55 | +To set the network location of a kube-apiserver that peers will use to proxy requests, use the |
| 56 | +`--peer-advertise-ip` and `--peer-advertise-port` command line arguments to kube-apiserver or specify |
| 57 | +these fields in the API server configuration file. |
| 58 | +If these flags are unspecified, peers will use the value from either `--advertise-address` or |
| 59 | +`--bind-address` command line argument to the kube-apiserver. If those too, are unset, the host's default interface is used. |
| 60 | + |
| 61 | +## Mixed version proxying |
| 62 | + |
| 63 | +When you enable mixed version proxying, the [aggregation layer](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) |
| 64 | +loads a special filter that does the following: |
| 65 | + |
| 66 | +* When a resource request reaches an API server that cannot serve that API (either because it is at a version pre-dating the introduction of the API or the API is turned off on the API server) the API server attempts to send the request to a peer API server that can serve the requested API. It does so by identifying API groups / versions / resources that the local server doesn't recognise, and tries to proxy those requests to a peer API server that is capable of handling the request. |
| 67 | +* If the peer API server fails to respond, the _source_ API server responds with 503("Service Unavailable") error. |
| 68 | + |
| 69 | +### How it works under the hood |
| 70 | + |
| 71 | +When an API Server receives a resource request, it first checks which API servers can serve the requested resource. This check happens using the internal [`StorageVersion` API]. |
| 72 | + |
| 73 | +* If the resource is known to the API server that received the request (ex: `GET /api/v1/pods/some-pod`), the request is handled locally. |
| 74 | + |
| 75 | +* If there is no internal `StorageVersion` object found for the requested resource (ex: `GET /my-api/v1/my-resource`) and the configured APIService specifies proxying to an extension API server, that proxying happens following the usual |
| 76 | +[flow](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) for |
| 77 | +extension APIs. |
| 78 | + |
| 79 | +* If a valid internal `StorageVersion` object is found for the requested resource (ex: `GET /batch/v1/jobs`) and the API server trying to handle the request (the _handling API server_) has the `batch` API disabled, then the _handling API server_fetches the peer API servers that do serve the relevant API group / version / resource (`api/v1/batch` in this case) using the information in the fetched `StorageVersion` object. The _handling API server_ then proxies the request to one of the matching peer kube-apiservers that are aware of the requested resource. |
| 80 | + * If there is no peer known for that API group / version / resource, the handling API server passes the request to its own handler chain which should eventually return a 404("Not Found") response. |
| 81 | + * If the handling API server has identified and selected a peer API server, but that peer fails |
| 82 | + to respond (for reasons such as network connectivity issues, or a data race between the request |
| 83 | + being received and a controller registering the peer's info into the control plane), then the handling |
| 84 | + API server responds with a 503 (“Service Unavailable”) error. |
0 commit comments