You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# HTTP proxy support in Azure Kubernetes Service (AKS)
13
13
14
-
Azure Kubernetes Service (AKS) clusters, whether deployed into a managed or custom virtual network, have certain outbound dependencies necessary to function properly. Previously, in environments requiring internet access to be routed through HTTP proxies, this was a problem. Nodes had no way of bootstrapping the configuration, environment variables, and certificates necessary to access internet services.
14
+
In this article, you learn how to configure Azure Kubernetes Service (AKS) clusters to use an HTTP proxy for outbound internet access.
15
15
16
-
This feature adds HTTP proxy support to AKS clusters, exposing a straightforward interface that cluster operators can use to secure AKS-required network traffic in proxy-dependent environments.
16
+
AKS clusters deployed into managed or custom virtual networks have certain outbound dependencies that are necessary to function properly, which created problems in environments requiring internet access to be routed through HTTP proxies. Nodes had no way of bootstrapping the configuration, environment variables, and certificates necessary to access internet services.
17
17
18
-
Both AKS nodes and Pods will be configured to use the HTTP proxy.
18
+
The HTTP proxy feature adds HTTP proxy support to AKS clusters, exposing a straightforward interface that you can use to secure AKS-required network traffic in proxy-dependent environments. With this feature, both AKS nodes and pods are configured to use the HTTP proxy. The feature also enables installation of a trusted certificate authority onto the nodes as part of bootstrapping a cluster. More complex solutions might require creating a chain of trust to establish secure communications across the network.
19
19
20
-
Some more complex solutions may require creating a chain of trust to establish secure communications across the network. The feature also enables installation of a trusted certificate authority onto the nodes as part of bootstrapping a cluster.
21
-
22
-
## Limitations and other details
20
+
## Limitations and considerations
23
21
24
22
The following scenarios are **not** supported:
25
23
26
-
- Different proxy configurations per node pool
27
-
- User/Password authentication
28
-
- Custom CAs for API server communication
29
-
- Windows-based clusters
30
-
- Node pools using Virtual Machine Availability Sets (VMAS)
31
-
- Using * as wildcard attached to a domain suffix for noProxy
24
+
* Different proxy configurations per node pool
25
+
* User/Password authentication
26
+
* Custom certificate authorities (CAs) for API server communication
27
+
* Windows-based clusters
28
+
* Node pools using Virtual Machine Availability Sets (VMAS)
29
+
* Using * as wildcard attached to a domain suffix for noProxy
32
30
33
-
By default, *httpProxy*, *httpsProxy*, and *trustedCa* have no value.
31
+
`httpProxy`, `httpsProxy`, and `trustedCa` have no value by default. Pods are injected with the following environment variables:
34
32
35
-
The Pods will be injected with the following environment variables:
36
-
-`HTTP_PROXY`
37
-
-`http_proxy`
38
-
-`HTTPS_PROXY`
39
-
-`https_proxy`
40
-
-`NO_PROXY`
41
-
-`no_proxy`
33
+
*`HTTP_PROXY`
34
+
*`http_proxy`
35
+
*`HTTPS_PROXY`
36
+
*`https_proxy`
37
+
*`NO_PROXY`
38
+
*`no_proxy`
42
39
43
-
To disable the injection of the proxy environment variables the Pod should be annotated with:`"kubernetes.azure.com/no-http-proxy-vars":"true"`
40
+
To disable the injection of the proxy environment variables, you need to annotate the Pod with `"kubernetes.azure.com/no-http-proxy-vars":"true"`.
44
41
45
-
## Prerequisites
42
+
## Before you begin
46
43
47
-
The latest version of the Azure CLI. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
44
+
* You need the latest version of the Azure CLI. Run `az --version` to find the version, and run `az upgrade` to upgrade the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
45
+
*[Check for available AKS cluster upgrades](./upgrade-aks-cluster.md#check-for-available-aks-cluster-upgrades) to ensure you're running the latest version of AKS. If you need to upgrade, see [Upgrade an AKS cluster](./upgrade-aks-cluster.md#upgrade-an-aks-cluster).
46
+
* The OS files required for proxy configuration updates can only be updated during the node image upgrade process. After configuring the proxy, you must upgrade the node image to apply the changes. For more information, see [Upgrade AKS node images](#upgrade-aks-node-images).
48
47
49
-
## Configuring an HTTP proxy using the Azure CLI
48
+
## Configure an HTTP proxy using the Azure CLI
50
49
51
-
Using AKS with an HTTP proxy is done at cluster creation, using the [az aks create][az-aks-create] command and passing in configuration as a JSON file.
50
+
You can configure an AKS cluster with an HTTP proxy during cluster creation using the [`az aks create`][az-aks-create] command and passing in configuration as a JSON file.
52
51
53
52
The schema for the config file looks like this:
54
53
@@ -64,14 +63,14 @@ The schema for the config file looks like this:
64
63
```
65
64
66
65
*`httpProxy`: A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be `http`.
67
-
*`httpsProxy`: A proxy URL to use for creating HTTPS connections outside the cluster. If this isn't specified, then `httpProxy` is used for both HTTP and HTTPS connections.
68
-
*`noProxy`: A list of destination domain names, domains, IP addresses or other network CIDRs to exclude proxying.
66
+
*`httpsProxy`: A proxy URL to use for creating HTTPS connections outside the cluster. If not specified, then `httpProxy` is used for both HTTP and HTTPS connections.
67
+
*`noProxy`: A list of destination domain names, domains, IP addresses, or other network CIDRs to exclude proxying.
69
68
*`trustedCa`: A string containing the `base64 encoded` alternative CA certificate content. Currently only the `PEM` format is supported.
70
69
71
70
> [!IMPORTANT]
72
71
> For compatibility with Go-based components that are part of the Kubernetes system, the certificate **must** support `Subject Alternative Names(SANs)` instead of the deprecated Common Name certs.
73
72
>
74
-
> There are differences in applications on how to comply with the environment variable `http_proxy`, `https_proxy`, and `no_proxy`. Curl and Python don't support CIDR in `no_proxy`, Ruby does.
73
+
> There are differences in applications on how to comply with the environment variable `http_proxy`, `https_proxy`, and `no_proxy`. Curl and Python don't support CIDR in `no_proxy`, but Ruby does.
75
74
76
75
Example input:
77
76
@@ -90,17 +89,15 @@ Example input:
90
89
}
91
90
```
92
91
93
-
Create a file and provide values for *httpProxy*, *httpsProxy*, and *noProxy*. If your environment requires it, provide a value for *trustedCa*. Next, deploy a cluster, passing in your filename using the `http-proxy-config`flag.
92
+
Create a file and provide values for `httpProxy`, `httpsProxy`, and `noProxy`. If your environment requires it, provide a value for `trustedCa`. Next, you can deploy the cluster using the [`az aks create`][az-aks-create] command with the `--http-proxy-config`parameter set to the file you created. Your cluster should initialize with the HTTP proxy configured on the nodes.
94
93
95
-
```azurecli
96
-
az aks create -n $clusterName -g $resourceGroup --http-proxy-config aks-proxy-config.json
94
+
```azurecli-interactive
95
+
az aks create --name $clusterName --resource-group $resourceGroup --http-proxy-config aks-proxy-config.json
97
96
```
98
97
99
-
Your cluster will initialize with the HTTP proxy configured on the nodes.
100
-
101
-
## Configuring an HTTP proxy using Azure Resource Manager (ARM) templates
98
+
## Configure an HTTP proxy using an Azure Resource Manager (ARM) template
102
99
103
-
Deploying an AKS cluster with an HTTP proxy configured using an ARM template is straightforward. The same schema used for CLI deployment exists in the `Microsoft.ContainerService/managedClusters` definition under properties:
100
+
You can deploy an AKS cluster with an HTTP proxy using an ARM template. The same schema used for CLI deployment exists in the `Microsoft.ContainerService/managedClusters` definition under `"properties"`, as shown in the following example:
104
101
105
102
```json
106
103
"properties": {
@@ -116,45 +113,46 @@ Deploying an AKS cluster with an HTTP proxy configured using an ARM template is
116
113
}
117
114
```
118
115
119
-
In your template, provide values for *httpProxy*, *httpsProxy*, and *noProxy*. If necessary, provide a value for *trustedCa*. Deploy the template, and your cluster should initialize with your HTTP proxy configured on the nodes.
116
+
In your template, provide values for `httpProxy`, `httpsProxy`, and `noProxy`. If necessary, provide a value for `trustedCa`. Next, you can deploy the template. Your cluster should initialize with your HTTP proxy configured on the nodes.
120
117
121
-
## Updating Proxy configurations
118
+
## Update proxy configuration
122
119
123
120
> [!NOTE]
124
-
> If switching to a new proxy, the new proxy must already exist for the update to be successful. Then, after the upgrade is completed the old proxy can be deleted.
121
+
> If switching to a new proxy, the new proxy must already exist for the update to be successful. After the upgrade is completed, you can delete the old proxy.
125
122
126
-
Values for *httpProxy*, *httpsProxy*, *trustedCa* and *NoProxy*can be changed and applied to the cluster with the [az aks update][az-aks-update] command. An aks update for *httpProxy*, *httpsProxy*, and/or *NoProxy* will automatically inject new environment variables into pods with the new *httpProxy*, *httpsProxy*, or *NoProxy* values. Pods must be rotated for the apps to pick it up, because the environment variable values are injected at the Pod creating by a mutating admission webhook. For components under kubernetes, like containerd and the node itself, this won't take effect until a node image upgrade is performed.
123
+
You can update the proxy configuration on your cluster using the [`az aks update`][az-aks-update] command with the `--http-proxy-config` parameter set to a new JSON file with updated values for `httpProxy`, `httpsProxy`, `noProxy`, and`trustedCa` if necessary. The update injects new environment variables into pods with the new `httpProxy`, `httpsProxy`, or `noProxy` values. Pods must be rotated for the apps to pick it up, because the environment variable values are injected by a mutating admission webhook. For components under Kubernetes, like containerd and the node itself, this doesn't take effect until a node image upgrade is performed.
127
124
128
-
For example, assuming a new file has been created with the base64 encoded string of the new CA cert called *aks-proxy-config-2.json*, the following action updates the cluster. Or, you need to add new endpoint urls for your applications to No Proxy:
125
+
For example, let's say you created a new file with the base64 encoded string of the new CA cert called *aks-proxy-config-2.json*. You can update the proxy configuration on your cluster with the following command:
129
126
130
-
```azurecli
131
-
az aks update -n $clusterName -g $resourceGroup --http-proxy-config aks-proxy-config-2.json
127
+
```azurecli-interactive
128
+
az aks update --name $clusterName --resource-group $resourceGroup --http-proxy-config aks-proxy-config-2.json
132
129
```
133
130
131
+
## Upgrade AKS node images
132
+
133
+
After configuring the proxy, you must upgrade the node image to apply the changes. The node image upgrade process is the only way to update the OS files required for proxy configuration updates. The node image upgrade process is a rolling upgrade that updates the OS image on each node in the node pool. The AKS control plane handles the upgrade process, which is nondisruptive to running applications.
134
+
135
+
To upgrade AKS node images, see [Upgrade Azure Kubernetes Service (AKS) node images](./node-image-upgrade.md).
136
+
134
137
## Monitoring add-on configuration
135
138
136
-
The HTTP proxy with the Monitoring add-on supports the following configurations:
139
+
HTTP proxy with the monitoring add-on supports the following configurations:
137
140
138
-
- Outbound proxy without authentication
139
-
- Outbound proxy with username & password authentication
140
-
- Outbound proxy with trusted cert for Log Analytics endpoint
141
+
* Outbound proxy without authentication
142
+
* Outbound proxy with username & password authentication
143
+
* Outbound proxy with trusted cert for Log Analytics endpoint
141
144
142
145
The following configurations aren't supported:
143
146
144
-
- The Custom Metrics and Recommended Alerts features aren't supported when you use a proxy with trusted certificates
147
+
*Custom Metrics and Recommended Alerts features when using a proxy with trusted certificates
145
148
146
149
## Next steps
147
150
148
-
For more information regarding the network requirements of AKS clusters, see [control egress traffic for cluster nodes in AKS][aks-egress].
151
+
For more information regarding the network requirements of AKS clusters, see [Control egress traffic for cluster nodes in AKS][aks-egress].
0 commit comments