Skip to content

Commit 0cb27ed

Browse files
author
Steven Smith
committed
Adds procedure and work for adding a hint file to the node IP configuration
1 parent d3761cb commit 0cb27ed

File tree

4 files changed

+105
-56
lines changed

4 files changed

+105
-56
lines changed

‎installing/installing_bare_metal_ipi/ipi-install-installation-workflow.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ include::modules/ipi-install-configuring-the-raid.adoc[leveloffset=+2]
7777

7878
include::modules/ipi-install-creating-a-disconnected-registry.adoc[leveloffset=+1]
7979

80+
8081
[discrete]
8182
[id="prerequisites_ipi-disconnected-registry"]
8283
=== Prerequisites

‎modules/nw-how-nw-iface-selected.adoc

Lines changed: 5 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -8,66 +8,15 @@ endif::[]
88
[id="nw-how-nw-iface-selected_{context}"]
99
= How the network interface is selected
1010

11-
For installations on bare metal or with virtual machines that have more than one network interface controller (NIC), the NIC that {product-title} uses for communication with the Kubernetes API server is determined by the `nodeip-configuration.service` service unit that is run by systemd when the node boots.
12-
The service iterates through the network interfaces on the node and the first network interface that is configured with a subnet than can host the IP address for the API server is selected for {product-title} communication.
11+
For installations on bare metal or with virtual machines that have more than one network interface controller (NIC), the NIC that {product-title} uses for communication with the Kubernetes API server is determined by the `nodeip-configuration.service` service unit that is run by systemd when the node boots. The `nodeip-configuration.service` selects the IP from the interface associated with the default route.
1312

14-
After the `nodeip-configuration.service` service determines the correct NIC, the service creates the `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` file.
15-
The `20-nodenet.conf` file sets the `KUBELET_NODE_IP` environment variable to the IP address that the service selected.
13+
After the `nodeip-configuration.service` service determines the correct NIC, the service creates the `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` file. The `20-nodenet.conf` file sets the `KUBELET_NODE_IP` environment variable to the IP address that the service selected.
1614

17-
When the kubelet service starts, it reads the value of the environment variable from the `20-nodenet.conf` file and sets the IP address as the value to the `--node-ip` kubelet command-line argument.
18-
As a result, the kubelet service uses the selected IP address as the node IP address.
15+
When the kubelet service starts, it reads the value of the environment variable from the `20-nodenet.conf` file and sets the IP address as the value of the `--node-ip` kubelet command-line argument. As a result, the kubelet service uses the selected IP address as the node IP address.
1916

20-
If hardware or networking is reconfigured after installation, it is possible that the `nodeip-configuration.service` service can select a different NIC after a reboot.
21-
In some cases, you might be able to detect that a different NIC is selected by reviewing the `INTERNAL-IP` column in the output from the `oc get nodes -o wide` command.
17+
If hardware or networking is reconfigured after installation, or if there is a networking layout where the node IP should not come from the default route interface, it is possible for the `nodeip-configuration.service` service to select a different NIC after a reboot. In some cases, you might be able to detect that a different NIC is selected by reviewing the `INTERNAL-IP` column in the output from the `oc get nodes -o wide` command.
2218

23-
If network communication is disrupted or misconfigured because a different NIC is selected, one strategy for overriding the selection process is to set the correct IP address explicitly.
24-
The following list identifies the high-level steps and considerations:
25-
26-
* Create a shell script that determines the IP address to use for {product-title} communication. Have the script create a custom unit file such as `/etc/systemd/system/kubelet.service.d/98-nodenet-override.conf`. Use the custom unit file, `98-nodenet-override.conf`, to set the `KUBELET_NODE_IP` environment variable to the IP address.
27-
28-
* Do not overwrite the `/etc/systemd/system/kubelet.service.d/20-nodenet.conf` file. Specify a file name with a numerically higher value such as `98-nodenet-override.conf` in the same directory path. The goal is to have the custom unit file run after `20-nodenet.conf` and override the value of the environment variable.
29-
30-
* Create a machine config object with the shell script as a base64-encoded string and use the Machine Config Operator to deploy the script to the nodes at a file system path such as `/usr/local/bin/override-node-ip.sh`.
31-
32-
* Ensure that `systemctl daemon-reload` runs after the shell script runs. The simplest method is to specify `ExecStart=systemctl daemon-reload` in the machine config, as shown in the following sample.
33-
34-
.Sample machine config to override the network interface for kubelet
35-
[source,yaml,subs="attributes+"]
36-
----
37-
apiVersion: machineconfiguration.openshift.io/v1
38-
kind: MachineConfig
39-
metadata:
40-
labels:
41-
machineconfiguration.openshift.io/role: worker
42-
name: 98-nodenet-override
43-
spec:
44-
config:
45-
ignition:
46-
version: {ign-config-version}
47-
storage:
48-
files:
49-
- contents:
50-
source: data:text/plain;charset=utf-8;base64,<encoded_script>
51-
mode: 0755
52-
overwrite: true
53-
path: /usr/local/bin/override-node-ip.sh
54-
systemd:
55-
units:
56-
- contents: |
57-
[Unit]
58-
Description=Override node IP detection
59-
Wants=network-online.target
60-
Before=kubelet.service
61-
After=network-online.target
62-
[Service]
63-
Type=oneshot
64-
ExecStart=/usr/local/bin/override-node-ip.sh
65-
ExecStart=systemctl daemon-reload
66-
[Install]
67-
WantedBy=multi-user.target
68-
enabled: true
69-
name: nodenet-override.service
70-
----
19+
If network communication is disrupted or misconfigured because a different NIC is selected, you might receive the following error: `EtcdCertSignerControllerDegraded`. You can create a hint file that includes the `NODEIP_HINT` variable to override the default IP selection logic. For more information, see Optional: Overriding the default node IP selection logic.
7120

7221
// Link to info for creating a machine config.
7322

‎modules/overriding-default-node-ip-selection-logic.adoc

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
// This is included in the following assemblies:
2+
//
3+
// * troubleshooting-network-issues.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="overriding-default-node-ip-selection-logic_{context}"]
7+
= Optional: Overriding the default node IP selection logic
8+
9+
To override the default IP selection logic, you can create a hint file that includes the `NODEIP_HINT` variable to override the default IP selection logic. Creating a hint file allows you to select a specific node IP address from the interface in the subnet of the IP address specified in the `NODEIP_HINT` variable.
10+
11+
For example, if a node has two interfaces, `eth0` with an address of `10.0.0.10/24`, and `eth1` with an address of `192.0.2.5/24`, and the default route points to `eth0` (`10.0.0.10`),the node IP address would normally use the `10.0.0.10` IP address.
12+
13+
Users can configure the `NODEIP_HINT` variable to point at a known IP in the subnet, for example, a subnet gateway such as `192.0.2.1` so that the other subnet, `192.0.2.0/24`, is selected. As a result, the `192.0.2.5` IP address on `eth1` is used for the node.
14+
15+
The following procedure shows how to override the default node IP selection logic.
16+
17+
.Procedure
18+
19+
. Add a hint file to your your `/etc/default/nodeip-configuration` file, for example:
20+
+
21+
[source,text]
22+
----
23+
NODEIP_HINT=192.0.2.1
24+
----
25+
+
26+
[IMPORTANT]
27+
====
28+
* Do not use the exact IP address of a node as a hint, for example, `192.0.2.5`. Using the exact IP address of a node causes the node using the hint IP address to fail to configure correctly.
29+
* The IP address in the hint file is only used to determine the correct subnet. It will not receive traffic as a result of appearing in the hint file.
30+
====
31+
32+
. Generate the `base-64` encoded content by running the following command:
33+
+
34+
[source,terminal]
35+
----
36+
$ echo 'NODEIP_HINT=192.0.2.1' | base64
37+
----
38+
+
39+
.Example output
40+
+
41+
[source,terminal]
42+
----
43+
Tk9ERUlQX0hJTlQ9MTkyLjAuMCxxxx==
44+
----
45+
46+
. Activate the hint by creating a machine config manifest for both `master` and `worker` roles before deploying the cluster:
47+
+
48+
.Master machine config manifest
49+
[source,yaml]
50+
----
51+
apiVersion: machineconfiguration.openshift.io/v1
52+
kind: MachineConfig
53+
metadata:
54+
labels:
55+
machineconfiguration.openshift.io/role: master
56+
name: 99-nodeip-hint-master
57+
spec:
58+
config:
59+
ignition:
60+
version: 3.2.0
61+
storage:
62+
files:
63+
- contents:
64+
source: data:text/plain;charset=utf-8;base64, <encoded_content> <1>
65+
mode: 0644
66+
overwrite: true
67+
path: /etc/default/nodeip-configuration
68+
----
69+
+
70+
<1> Replace `<encoded_contents>` with the base64-encoded content of the `/etc/default/nodeip-configuration` file, for example, `Tk9ERUlQX0hJTlQ9MTkyLjAuMCxxxx==`.
71+
+
72+
.Worker machine config manifest
73+
[source,yaml]
74+
----
75+
apiVersion: machineconfiguration.openshift.io/v1
76+
kind: MachineConfig
77+
metadata:
78+
labels:
79+
machineconfiguration.openshift.io/role: worker
80+
name: 99-nodeip-hint-worker
81+
spec:
82+
config:
83+
ignition:
84+
version: 3.2.0
85+
storage:
86+
files:
87+
- contents:
88+
source: data:text/plain;charset=utf-8;base64, <encoded_content> <1>
89+
mode: 0644
90+
overwrite: true
91+
path: /etc/default/nodeip-configuration
92+
----
93+
<1> Replace `<encoded_contents>` with the base64-encoded content of the `/etc/default/nodeip-configuration` file, for example, `Tk9ERUlQX0hJTlQ9MTkyLjAuMCxxxx==`.
94+
95+
. Save the manifest to the directory where you store your cluster configuration, for example, `~/clusterconfigs`.
96+
97+
. Deploy the cluster.

‎support/troubleshooting/troubleshooting-network-issues.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ toc::[]
99
// How the network interface is selected
1010
include::modules/nw-how-nw-iface-selected.adoc[leveloffset=+1]
1111

12+
include::modules/overriding-default-node-ip-selection-logic.adoc[leveloffset=+2]
13+
1214
// Troubleshooting OVS issues
1315
include::modules/nw-troubleshoot-ovs.adoc[leveloffset=+1]
1416

0 commit comments

Comments
 (0)