Skip to content

Commit 7dd8ab2

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #40204 from xuzhenglun/blog
add blog for feature ServiceNodePortStaticSubrange
2 parents a6ca029 + 3292ae8 commit 7dd8ab2

File tree

1 file changed

+173
-0
lines changed

1 file changed

+173
-0
lines changed

‎content/en/blog/_posts/2023-05-11-nodeport-dynamic-and-static-allocation.md

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.27: Avoid Collisions Assigning Ports to NodePort Services"
4+
date: 2023-05-11
5+
slug: nodeport-dynamic-and-static-allocation
6+
---
7+
8+
**Author:** Xu Zhenglun (Alibaba)
9+
10+
In Kubernetes, a Service can be used to provide a unified traffic endpoint for
11+
applications running on a set of Pods. Clients can use the virtual IP address (or _VIP_) provided
12+
by the Service for access, and Kubernetes provides load balancing for traffic accessing
13+
different back-end Pods, but a ClusterIP type of Service is limited to providing access to
14+
nodes within the cluster, while traffic from outside the cluster cannot be routed.
15+
One way to solve this problem is to use a `type: NodePort` Service, which sets up a mapping
16+
to a specific port of all nodes in the cluster, thus redirecting traffic from the
17+
outside to the inside of the cluster.
18+
19+
## How Kubernetes allocates node ports to Services?
20+
21+
When a `type: NodePort` Service is created, its corresponding port(s) are allocated in one
22+
of two ways:
23+
24+
- **Dynamic** : If the Service type is `NodePort` and you do not set a `nodePort`
25+
value explicitly in the `spec` for that Service, the Kubernetes control plane will
26+
automatically allocate an unused port to it at creation time.
27+
28+
- **Static** : In addition to the dynamic auto-assignment described above, you can also
29+
explicitly assign a port that is within the nodeport port range configuration.
30+
31+
The value of `nodePort` that you manually assign must be unique across the whole cluster.
32+
Attempting to create a Service of `type: NodePort` where you explicitly specify a node port that
33+
was already allocated results in an error.
34+
35+
## Why do you need to reserve ports of NodePort Service?
36+
Sometimes, you may want to have a NodePort Service running on well-known ports
37+
so that other components and users inside o r outside the cluster can use them.
38+
39+
In some complex cluster deployments with a mix of Kubernetes nodes and other servers on the same network,
40+
it may be necessary to use some pre-defined ports for communication. In particular, some fundamental
41+
components cannot rely on the VIPs that back `type: LoadBalancer` Services
42+
because the virtual IP address mapping implementation for that cluster also relies on
43+
these foundational components.
44+
45+
Now suppose you need to expose a Minio object storage service on Kubernetes to clients
46+
running outside the Kubernetes cluster, and the agreed port is `30009`, we need to
47+
create a Service as follows:
48+
49+
```yaml
50+
apiVersion: v1
51+
kind: Service
52+
metadata:
53+
name: minio
54+
spec:
55+
ports:
56+
- name: api
57+
nodePort: 30009
58+
port: 9000
59+
protocol: TCP
60+
targetPort: 9000
61+
selector:
62+
app: minio
63+
type: NodePort
64+
```
65+
66+
However, as mentioned before, if the port (30009) required for the `minio` Service is not reserved,
67+
and another `type: NodePort` (or possibly `type: LoadBalancer`) Service is created and dynamically
68+
allocated before or concurrently with the `minio` Service, TCP port 30009 might be allocated to that
69+
other Service; if so, creation of the `minio` Service will fail due to a node port collision.
70+
71+
## How can you avoid NodePort Service port conflicts?
72+
Kubernetes 1.24 introduced changes for `type: ClusterIP` Services, dividing the CIDR range for cluster
73+
IP addresses into two blocks that use different allocation policies to [reduce the risk of conflicts](/docs/reference/networking/virtual-ips/#avoiding-collisions).
74+
In Kubernetes 1.27, as an alpha feature, you can adopt a similar policy for `type: NodePort` Servies.
75+
You can enable a new [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
76+
`ServiceNodePortStaticSubrange`. Turning this on allows you to use a different port allocation strategy
77+
for `type: NodePort` Services, and reduce the risk of collision.
78+
79+
The port range for `NodePort` will be divided, based on the formula `min(max(16, nodeport-size / 32), 128)`.
80+
The outcome of the formula will be a number between 16 and 128, with a step size that increases as the
81+
size of the nodeport range increases. The outcome of the formula determine that the size of static port
82+
range. When the port range is less than 16, the size of static port range will be set to 0,
83+
which means that all ports will be dynamically allocated.
84+
85+
Dynamic port assignment will use the upper band by default, once this has been exhausted it will use the lower range.
86+
This will allow users to use static allocations on the lower band with a low risk of collision.
87+
88+
## Examples
89+
90+
### default range: 30000-32767
91+
| Range properties | Values |
92+
|-------------------------|-------------------------------------------------------|
93+
| service-node-port-range | 30000-32767 |
94+
| Band Offset | &ensp; `min(max(16, 2768/32), 128)` <br>= `min(max(16, 86), 128)` <br>= `min(86, 128)` <br>= 86 |
95+
| Static band start | 30000 |
96+
| Static band end | 30085 |
97+
| Dynamic band start | 30086 |
98+
| Dynamic band end | 32767 |
99+
100+
{{< mermaid >}}
101+
pie showData
102+
title 30000-32767
103+
"Static" : 86
104+
"Dynamic" : 2682
105+
{{< /mermaid >}}
106+
107+
### very small range: 30000-30015
108+
| Range properties | Values |
109+
|-------------------------|-------------------------------------------------------|
110+
| service-node-port-range | 30000-30015 |
111+
| Band Offset | 0 |
112+
| Static band start | - |
113+
| Static band end | - |
114+
| Dynamic band start | 30000 |
115+
| Dynamic band end | 30015 |
116+
117+
{{< mermaid >}}
118+
pie showData
119+
title 30000-30015
120+
"Static" : 0
121+
"Dynamic" : 16
122+
{{< /mermaid >}}
123+
124+
### small(lower boundary) range: 30000-30127
125+
| Range properties | Values |
126+
|-------------------------|-------------------------------------------------------|
127+
| service-node-port-range | 30000-30127 |
128+
| Band Offset | &ensp; `min(max(16, 128/32), 128)` <br>= `min(max(16, 4), 128)` <br>= `min(16, 128)` <br>= 16 |
129+
| Static band start | 30000 |
130+
| Static band end | 30015 |
131+
| Dynamic band start | 30016 |
132+
| Dynamic band end | 30127 |
133+
134+
{{< mermaid >}}
135+
pie showData
136+
title 30000-30127
137+
"Static" : 16
138+
"Dynamic" : 112
139+
{{< /mermaid >}}
140+
141+
### large(upper boundary) range: 30000-34095
142+
| Range properties | Values |
143+
|-------------------------|-------------------------------------------------------|
144+
| service-node-port-range | 30000-34095 |
145+
| Band Offset | &ensp; `min(max(16, 4096/32), 128)` <br>= `min(max(16, 128), 128)` <br>= `min(128, 128)` <br>= 128 |
146+
| Static band start | 30000 |
147+
| Static band end | 30127 |
148+
| Dynamic band start | 30128 |
149+
| Dynamic band end | 34095 |
150+
151+
{{< mermaid >}}
152+
pie showData
153+
title 30000-34095
154+
"Static" : 128
155+
"Dynamic" : 3968
156+
{{< /mermaid >}}
157+
158+
### very large range: 30000-38191
159+
| Range properties | Values |
160+
|-------------------------|-------------------------------------------------------|
161+
| service-node-port-range | 30000-38191 |
162+
| Band Offset | &ensp; `min(max(16, 8192/32), 128)` <br>= `min(max(16, 256), 128)` <br>= `min(256, 128)` <br>= 128 |
163+
| Static band start | 30000 |
164+
| Static band end | 30127 |
165+
| Dynamic band start | 30128 |
166+
| Dynamic band end | 38191 |
167+
168+
{{< mermaid >}}
169+
pie showData
170+
title 30000-38191
171+
"Static" : 128
172+
"Dynamic" : 8064
173+
{{< /mermaid >}}

0 commit comments

Comments
 (0)