Skip to content

Commit ed8d574

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #32190 from tengqm/fix-nodelocaldns
Reformat the node local DNS cache page
2 parents 6cffdff + 29f5e89 commit ed8d574

File tree

1 file changed

+81
-38
lines changed

1 file changed

+81
-38
lines changed

‎content/en/docs/tasks/administer-cluster/nodelocaldns.md

Lines changed: 81 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -8,36 +8,49 @@ content_type: task
88
---
99

1010
<!-- overview -->
11-
{{< feature-state for_k8s_version="v1.18" state="stable" >}}
12-
This page provides an overview of NodeLocal DNSCache feature in Kubernetes.
1311

12+
{{< feature-state for_k8s_version="v1.18" state="stable" >}}
1413

14+
This page provides an overview of NodeLocal DNSCache feature in Kubernetes.
1515

1616
## {{% heading "prerequisites" %}}
1717

18-
19-
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
20-
18+
{{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
2119

2220
<!-- steps -->
2321

2422
## Introduction
2523

26-
NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet. In today's architecture, Pods in ClusterFirst DNS mode reach out to a kube-dns serviceIP for DNS queries. This is translated to a kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy. With this new architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking. The local caching agent will query kube-dns service for cache misses of cluster hostnames(cluster.local suffix by default).
27-
24+
NodeLocal DNSCache improves Cluster DNS performance by running a DNS caching agent
25+
on cluster nodes as a DaemonSet. In today's architecture, Pods in 'ClusterFirst' DNS mode
26+
reach out to a kube-dns `serviceIP` for DNS queries. This is translated to a
27+
kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy.
28+
With this new architecture, Pods will reach out to the DNS caching agent
29+
running on the same node, thereby avoiding iptables DNAT rules and connection tracking.
30+
The local caching agent will query kube-dns service for cache misses of cluster
31+
hostnames ("`cluster.local`" suffix by default).
2832

2933
## Motivation
3034

31-
* With the current DNS architecture, it is possible that Pods with the highest DNS QPS have to reach out to a different node, if there is no local kube-dns/CoreDNS instance.
32-
Having a local cache will help improve the latency in such scenarios.
35+
* With the current DNS architecture, it is possible that Pods with the highest DNS QPS
36+
have to reach out to a different node, if there is no local kube-dns/CoreDNS instance.
37+
Having a local cache will help improve the latency in such scenarios.
3338

34-
* Skipping iptables DNAT and connection tracking will help reduce [conntrack races](https://github.com/kubernetes/kubernetes/issues/56903) and avoid UDP DNS entries filling up conntrack table.
39+
* Skipping iptables DNAT and connection tracking will help reduce
40+
[conntrack races](https://github.com/kubernetes/kubernetes/issues/56903)
41+
and avoid UDP DNS entries filling up conntrack table.
3542

36-
* Connections from local caching agent to kube-dns service can be upgraded to TCP. TCP conntrack entries will be removed on connection close in contrast with UDP entries that have to timeout ([default](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt) `nf_conntrack_udp_timeout` is 30 seconds)
43+
* Connections from local caching agent to kube-dns service can be upgraded to TCP.
44+
TCP conntrack entries will be removed on connection close in contrast with
45+
UDP entries that have to timeout
46+
([default](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt)
47+
`nf_conntrack_udp_timeout` is 30 seconds)
3748

38-
* Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout). Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed.
49+
* Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to
50+
dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout).
51+
Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed.
3952

40-
* Metrics & visibility into dns requests at a node level.
53+
* Metrics & visibility into DNS requests at a node level.
4154

4255
* Negative caching can be re-enabled, thereby reducing number of queries to kube-dns service.
4356

@@ -49,48 +62,75 @@ This is the path followed by DNS Queries after NodeLocal DNSCache is enabled:
4962
{{< figure src="/images/docs/nodelocaldns.svg" alt="NodeLocal DNSCache flow" title="Nodelocal DNSCache flow" caption="This image shows how NodeLocal DNSCache handles DNS queries." class="diagram-medium" >}}
5063

5164
## Configuration
52-
{{< note >}} The local listen IP address for NodeLocal DNSCache can be any address that can be guaranteed to not collide with any existing IP in your cluster. It's recommended to use an address with a local scope, per example, from the link-local range 169.254.0.0/16 for IPv4 or from the Unique Local Address range in IPv6 fd00::/8.
65+
66+
{{< note >}}
67+
The local listen IP address for NodeLocal DNSCache can be any address that
68+
can be guaranteed to not collide with any existing IP in your cluster.
69+
It's recommended to use an address with a local scope, per example,
70+
from the 'link-local' range '169.254.0.0/16' for IPv4 or from the
71+
'Unique Local Address' range in IPv6 'fd00::/8'.
5372
{{< /note >}}
5473

5574
This feature can be enabled using the following steps:
5675

57-
* Prepare a manifest similar to the sample [`nodelocaldns.yaml`](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml) and save it as `nodelocaldns.yaml.`
58-
* If using IPv6, the CoreDNS configuration file need to enclose all the IPv6 addresses into square brackets if used in IP:Port format.
59-
If you are using the sample manifest from the previous point, this will require to modify [the configuration line L70](https://github.com/kubernetes/kubernetes/blob/b2ecd1b3a3192fbbe2b9e348e095326f51dc43dd/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L70) like this `health [__PILLAR__LOCAL__DNS__]:8080`
60-
* Substitute the variables in the manifest with the right values:
76+
* Prepare a manifest similar to the sample
77+
[`nodelocaldns.yaml`](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml)
78+
and save it as `nodelocaldns.yaml.`
6179

62-
* kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
80+
* If using IPv6, the CoreDNS configuration file need to enclose all the IPv6 addresses
81+
into square brackets if used in 'IP:Port' format.
82+
If you are using the sample manifest from the previous point, this will require to modify
83+
[the configuration line L70](https://github.com/kubernetes/kubernetes/blob/b2ecd1b3a3192fbbe2b9e348e095326f51dc43dd/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L70)
84+
like this: "`health [__PILLAR__LOCAL__DNS__]:8080`"
6385

64-
* domain=`<cluster-domain>`
86+
* Substitute the variables in the manifest with the right values:
6587

66-
* localdns=`<node-local-address>`
88+
```shell
89+
kubedns=`kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}`
90+
domain=<cluster-domain>
91+
localdns=<node-local-address>
92+
```
6793

68-
`<cluster-domain>` is "cluster.local" by default. `<node-local-address>` is the local listen IP address chosen for NodeLocal DNSCache.
94+
`<cluster-domain>` is "`cluster.local`" by default. `<node-local-address>` is the
95+
local listen IP address chosen for NodeLocal DNSCache.
6996

70-
* If kube-proxy is running in IPTABLES mode:
97+
* If kube-proxy is running in IPTABLES mode:
7198

72-
``` bash
73-
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
74-
```
99+
``` bash
100+
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
101+
```
75102

76-
`__PILLAR__CLUSTER__DNS__` and `__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
77-
In this mode, node-local-dns pods listen on both the kube-dns service IP as well as `<node-local-address>`, so pods can lookup DNS records using either IP address.
103+
`__PILLAR__CLUSTER__DNS__` and `__PILLAR__UPSTREAM__SERVERS__` will be populated by
104+
the `node-local-dns` pods.
105+
In this mode, the `node-local-dns` pods listen on both the kube-dns service IP
106+
as well as `<node-local-address>`, so pods can lookup DNS records using either IP address.
78107

79108
* If kube-proxy is running in IPVS mode:
80109

81110
``` bash
82-
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
111+
sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml
83112
```
84-
In this mode, node-local-dns pods listen only on `<node-local-address>`. The node-local-dns interface cannot bind the kube-dns cluster IP since the interface used for IPVS loadbalancing already uses this address.
85-
`__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
113+
114+
In this mode, the `node-local-dns` pods listen only on `<node-local-address>`.
115+
The `node-local-dns` interface cannot bind the kube-dns cluster IP since the
116+
interface used for IPVS loadbalancing already uses this address.
117+
`__PILLAR__UPSTREAM__SERVERS__` will be populated by the node-local-dns pods.
86118

87119
* Run `kubectl create -f nodelocaldns.yaml`
88-
* If using kube-proxy in IPVS mode, `--cluster-dns` flag to kubelet needs to be modified to use `<node-local-address>` that NodeLocal DNSCache is listening on.
89-
Otherwise, there is no need to modify the value of the `--cluster-dns` flag, since NodeLocal DNSCache listens on both the kube-dns service IP as well as `<node-local-address>`.
90120

91-
Once enabled, node-local-dns Pods will run in the kube-system namespace on each of the cluster nodes. This Pod runs [CoreDNS](https://github.com/coredns/coredns) in cache mode, so all CoreDNS metrics exposed by the different plugins will be available on a per-node basis.
121+
* If using kube-proxy in IPVS mode, `--cluster-dns` flag to kubelet needs to be modified
122+
to use `<node-local-address>` that NodeLocal DNSCache is listening on.
123+
Otherwise, there is no need to modify the value of the `--cluster-dns` flag,
124+
since NodeLocal DNSCache listens on both the kube-dns service IP as well as
125+
`<node-local-address>`.
126+
127+
Once enabled, the `node-local-dns` Pods will run in the `kube-system` namespace
128+
on each of the cluster nodes. This Pod runs [CoreDNS](https://github.com/coredns/coredns)
129+
in cache mode, so all CoreDNS metrics exposed by the different plugins will
130+
be available on a per-node basis.
92131

93-
You can disable this feature by removing the DaemonSet, using `kubectl delete -f <manifest>` . You should also revert any changes you made to the kubelet configuration.
132+
You can disable this feature by removing the DaemonSet, using `kubectl delete -f <manifest>`.
133+
You should also revert any changes you made to the kubelet configuration.
94134

95135
## StubDomains and Upstream server Configuration
96136

@@ -103,7 +143,9 @@ In those cases, the `kube-dns` ConfigMap can be updated.
103143

104144
## Setting memory limits
105145

106-
node-local-dns pods use memory for storing cache entries and processing queries. Since they do not watch Kubernetes objects, the cluster size or the number of Services/Endpoints do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
146+
The `node-local-dns` Pods use memory for storing cache entries and processing queries.
147+
Since they do not watch Kubernetes objects, the cluster size or the number of Services/Endpoints
148+
do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
107149
From [CoreDNS docs](https://github.com/coredns/deployment/blob/master/kubernetes/Scaling_CoreDNS.md),
108150
> The default cache size is 10000 entries, which uses about 30 MB when completely filled.
109151

@@ -114,17 +156,18 @@ The number of concurrent queries is linked to the memory demand, because each ex
114156
goroutine used for handling a query requires an amount of memory. You can set an upper limit
115157
using the `max_concurrent` option in the forward plugin.
116158

117-
If a node-local-dns pod attempts to use more memory than is available (because of total system
159+
If a `node-local-dns` Pod attempts to use more memory than is available (because of total system
118160
resources, or because of a configured
119161
[resource limit](/docs/concepts/configuration/manage-resources-containers/)), the operating system
120162
may shut down that pod's container.
121163
If this happens, the container that is terminated (“OOMKilled”) does not clean up the custom
122164
packet filtering rules that it previously added during startup.
123-
The node-local-dns container should get restarted (since managed as part of a DaemonSet), but this
165+
The `node-local-dns` container should get restarted (since managed as part of a DaemonSet), but this
124166
will lead to a brief DNS downtime each time that the container fails: the packet filtering rules direct
125167
DNS queries to a local Pod that is unhealthy.
126168
127169
You can determine a suitable memory limit by running node-local-dns pods without a limit and
128170
measuring the peak usage. You can also set up and use a
129171
[VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler)
130172
in _recommender mode_, and then check its recommendations.
173+

0 commit comments

Comments
 (0)