Skip to content

Commit 1be0a16

Browse files
Merge pull request #32693 from mikemckiernan/fix-node-ip
BZ-1944001: specifying node-ip through MCO
2 parents f7cd4d1 + d81b249 commit 1be0a16

File tree

4 files changed

+91
-1
lines changed

4 files changed

+91
-1
lines changed

_topic_map.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,9 @@ Topics:
458458
- Name: Troubleshooting operating system issues
459459
File: troubleshooting-operating-system-issues
460460
Distros: openshift-enterprise,openshift-origin
461+
- Name: Troubleshooting network issues
462+
File: troubleshooting-network-issues
463+
Distros: openshift-enterprise,openshift-origin
461464
- Name: Troubleshooting Operator issues
462465
File: troubleshooting-operator-issues
463466
- Name: Investigating pod issues

modules/machine-config-overview.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ a machine config that is injected directly into the {product-title} installer pr
2424

2525
* Manual changes to nodes are strongly discouraged. If you need to decommission a node and start a new one, those direct changes would be lost.
2626

27-
* MCO is only supported for writing to files in `/etc` and `/var` directories, although there are symbolic links to some directories that can be writeable by being symbolically linked to one of those areas. The `/opt` directory is an example.
27+
* MCO is only supported for writing to files in `/etc` and `/var` directories, although there are symbolic links to some directories that can be writeable by being symbolically linked to one of those areas. The `/opt` and `/usr/local` directories are examples.
2828

2929
* Ignition is the configuration format used in MachineConfigs. See the link:https://coreos.github.io/ignition/configuration-v3_2/[Ignition Configuration Specification v3.2.0] for details.
3030

modules/nw-how-nw-iface-selected.adoc

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
// Applies to 4.6 and newer.
2+
3+
:ign-config-version: 3.1.0
4+
ifeval::[{product-version} > 4.6]
5+
:ign-config-version: 3.2.0
6+
endif::[]
7+
8+
[id="nw-how-nw-iface-selected_{context}"]
9+
= How the network interface is selected
10+
11+
For installations on bare metal or with virtual machines that have more than one network interface controller (NIC), the NIC that {product-title} uses for communication with the Kubernetes API server is determined by the `nodeip-configuration.service` service unit that is run by systemd when the node boots.
12+
The service iterates through the network interfaces on the node and the first network interface that is configured with a subnet than can host the IP address for the API server is selected for {product-title} communication.
13+
14+
After the `nodeip-configuration.service` service determines the correct NIC, the service creates the `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` file.
15+
The `20-nodenet.conf` file sets the `KUBELET_NODE_IP` environment variable to the IP address that the service selected.
16+
17+
When the kubelet service starts, it reads the value of the environment variable from the `20-nodenet.conf` file and sets the IP address as the value to the `--node-ip` kubelet command-line argument.
18+
As a result, the kubelet service uses the selected IP address as the node IP address.
19+
20+
If hardware or networking is reconfigured after installation, it is possible that the `nodeip-configuration.service` service can select a different NIC after a reboot.
21+
In some cases, you might be able to detect that a different NIC is selected by reviewing the `INTERNAL-IP` column in the output from the `oc get nodes -o wide` command.
22+
23+
If network communication is disrupted or misconfigured because a different NIC is selected, one strategy for overriding the selection process is to set the correct IP address explicitly.
24+
The following list identifies the high-level steps and considerations:
25+
26+
* Create a shell script that determines the IP address to use for {product-title} communication. Have the script create a custom unit file such as `/etc/systemd/system/kubelet.service.d/98-nodenet-override.conf`. Use the custom unit file, `98-nodenet-override.conf`, to set the `KUBELET_NODE_IP` environment variable to the IP address.
27+
28+
* Do not overwrite the `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` file. Specify a file name with a numerically higher value such as `98-nodenet-override.conf` in the same directory path. The goal is to have the custom unit file run after `20-nodenet.conf` and override the value of the environment variable.
29+
30+
* Create a machine config object with the shell script as a base64-encoded string and use the Machine Config Operator to deploy the script to the nodes at a file system path such as `/usr/local/bin/override-node-ip.sh`.
31+
32+
* Ensure that `systemctl daemon-reload` runs after the shell script runs. The simplest method is to specify `ExecStart=systemctl daemon-reload` in the machine config, as shown in the following sample.
33+
34+
.Sample machine config to override the network interface for kubelet
35+
[source,yaml,subs="attributes+"]
36+
----
37+
apiVersion: machineconfiguration.openshift.io/v1
38+
kind: MachineConfig
39+
metadata:
40+
labels:
41+
machineconfiguration.openshift.io/role: worker
42+
name: 98-nodenet-override
43+
spec:
44+
config:
45+
ignition:
46+
version: {ign-config-version}
47+
storage:
48+
files:
49+
- contents:
50+
source: data:text/plain;charset=utf-8;base64,<encoded_script>
51+
mode: 0755
52+
overwrite: true
53+
path: /usr/local/bin/override-node-ip.sh
54+
systemd:
55+
units:
56+
- contents: |
57+
[Unit]
58+
Description=Override node IP detection
59+
Wants=network-online.target
60+
Before=kubelet.service
61+
After=network-online.target
62+
[Service]
63+
Type=oneshot
64+
ExecStart=/usr/local/bin/override-node-ip.sh
65+
ExecStart=systemctl daemon-reload
66+
[Install]
67+
WantedBy=multi-user.target
68+
enabled: true
69+
name: nodenet-override.service
70+
71+
----
72+
73+
// Link to info for creating a machine config.
74+
75+
// Clear temporary attributes
76+
ifdef::ign-config-version[]
77+
:!ign-config-version:
78+
endif::[]
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[id="troubleshooting-network-issues"]
2+
= Troubleshooting network issues
3+
include::modules/common-attributes.adoc[]
4+
:context: troubleshooting-network-issues
5+
6+
toc::[]
7+
8+
// How the network interface is selected
9+
include::modules/nw-how-nw-iface-selected.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)