Skip to content

feat: update docs for NetworkManager #897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

feat: update docs for NetworkManager #897

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c875267
feat: update docs for NetworkManager
tserong Oct 10, 2025
3ab9c7b
WIP: add NetworkManager TODO items
tserong Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/install/harvester-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,10 +266,10 @@ DNS resolution is unavailable in the `after-install-chroot stage`, and the `name
```yaml
os:
after_install_chroot_commands:
- "rm -f /etc/resolv.conf && echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
- "echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf"
- "mkdir /usr/local/bin"
- "curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh"
- "rm -f /etc/resolv.conf && ln -s /var/run/netconfig/resolv.conf /etc/resolv.conf"
- "rm -f /etc/resolv.conf"
```


Expand Down
2 changes: 1 addition & 1 deletion docs/install/requirements.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Harvester nodes require the following port connections or inbound rules. Typical
| TCP | 9796 | Harvester management and compute nodes | Node-exporter |
| TCP | 30000-32767 | Harvester management and compute nodes | NodePort port range |
| TCP | 22 | Harvester management and compute nodes | sshd |
| UDP | 68 | Harvester management and compute nodes | Wicked |
| UDP | 68 | Harvester management and compute nodes | NetworkManager |
| TCP | 3260 | Harvester management and compute nodes | iscsid |

### Port Requirements for Integrating Harvester with Rancher
Expand Down
51 changes: 42 additions & 9 deletions docs/install/update-harvester-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,25 @@ This page describes how to edit some of the most-requested Harvester configurati

If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples will be `/oem/99_custom.yaml`. Please substitute the value if needed.

If you upgrade from a version before `v1.7.0`, any changes to network and DNS configuration need to be made in the `/oem/91_networkmanager.yaml` file. Please substitute the value if needed.

:::

## DNS servers

### Runtime change

1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
1. Edit `/etc/sysconfig/network/config` and update the following line. Use a space to separate DNS server addresses if there are multiple servers.
1. If the management interface _is not_ configured to use a VLAN, run the following command:

```
NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"
nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br
```

1. Update and reload the configuration with the following command:
1. If the management interface _is_ configured to use a VLAN, run the following commands. Replace `VLAN_ID` with the actal ID of the VLAN. If in doubt, run `nmcli con` to see the configured connections and devices.

```
netconfig update
nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.VLAN_ID
```

1. Confirm the file `/etc/resolv.conf` contains the correct DNS servers with the `cat` command:
Expand All @@ -59,24 +61,35 @@ Beginning with v1.1.2, the persistent name of the cloud-init file is `/oem/90_cu

When upgrading from an earlier version to `v1.1.2` or later, Harvester retains the old file name (`/oem/99_custom.yaml`) to avoid confusion. You can manually rename the file to `/oem/90_custom.yaml` if necessary.

If you have upgraded from a version before `v1.7.0`, the changes below need to be made in the `/oem/91_networkmanager.yaml` file.

1. Backup the elemental `cloud-init` file `/oem/90_custom.yaml` as follows:

```
cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes)
```

1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.initramfs[0].commands`. The `commands` array must contain a line to manipulate the `NETCONFIG_DNS_STATIC_SERVERS` config. Add the line if the line doesn't exist.
1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.network[0].commands`. The `commands` array must contain a line to manipulate the NetworkManager DNS configuration for the management interface. This is the exact same command used above when making the change at runtime. Add the line if the line doesn't exist.

The following example adds a line to change the `NETCONFIG_DNS_STATIC_SERVERS` config:
The following example adds a line to configure DNS servers when not using a VLAN on the management interface:

```
stages:
initramfs:
network:
- commands:
- nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br
```

The following example adds a line to configure DNS servers when using VLAN 2017 on the management interface:

```
stages:
network:
- commands:
- sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"/' /etc/sysconfig/network/config
- nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.2017
```

Replace the DNS server addresses and save the file. Harvester sets up new servers after rebooting.
Replace the DNS server addresses and VLAN ID if applicable and save the file. Harvester sets up new servers after rebooting.


## NTP servers
Expand Down Expand Up @@ -109,6 +122,21 @@ You can update the slave interfaces of Harvester's management bonding interface

### Runtime change

:::info DOCS TODO

Procedure is to add a new NM connection profile for the new interface, e.g.: `/etc/NetworkManager/system-connections`, with content:
```
[connection]
id=bond-slave-eno50
type=ethernet
interface-name=eno50
master=mgmt-bo
slave-type=bond
```
...then `nmcli con reload` and probably `nmcli device reapply mgmt-bo`.

:::

1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details.
1. Identify the interface names with the following command:

Expand Down Expand Up @@ -138,6 +166,11 @@ You can update the slave interfaces of Harvester's management bonding interface

### Configuration persistence

:::info DOCS TODO

Files as in above TODO, and note `/oem/91_networkmanager.yaml` for pre v1.7.0 systems

:::

1. Backup the elemental cloud-init file `/oem/90_custom.yaml` as follows:

Expand Down
6 changes: 6 additions & 0 deletions docs/logging/harvester-logging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ In Harvester's practice, the `Logging`, `Audit` and `Event` shares one architect
The Harvester logging infrastructure allows you to aggregate Harvester logs into an external service such as [Graylog](https://www.graylog.org), [Elasticsearch](https://www.elastic.co/elasticsearch/), [Splunk](https://www.splunk.com/), [Grafana Loki](https://grafana.com/oss/loki/) and others.

### Collected Logs
:::info DOCS TODO

Confirm this does NetworkManager logs and fix it if not.

:::

See below for a list logs that are collected:
- Logs from all cluster `Pods`
- Kernel logs from each `node`
Expand Down
36 changes: 24 additions & 12 deletions docs/networking/best-practice.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,28 +45,40 @@ Before any action is taken, it is important to collect the current network infor
Example: A NIC named `ens3` was added to the `mgmt-bo` bond interface.

```
- path: /etc/sysconfig/network/ifcfg-mgmt-bo
- path: /etc/NetworkManager/system-connections/bond-mgmt.nmconnection
permissions: 384
owner: 0
group: 0
content: |+
STARTMODE='onboot'
BONDING_MASTER='yes'
BOOTPROTO='none'
POST_UP_SCRIPT="wicked:setup_bond.sh"
BONDING_SLAVE_0='ens3'
BONDING_MODULE_OPTS='miimon=100 mode=active-backup '
DHCLIENT_SET_DEFAULT_ROUTE='no'
content: |-
[connection]
id=bond-mgmt
type=bond
interface-name=mgmt-bo
master=mgmt-br
slave-type=bridge

[ethernet]

[bond]
miimon=100
mode=active-backup

[bridge-port]
vlans=1 pvid untagged
encoding: ""
ownerstring: ""

- path: /etc/sysconfig/network/ifcfg-ens3
- path: /etc/NetworkManager/system-connections/bond-slave-ens3.nmconnection
permissions: 384
owner: 0
group: 0
content: |
STARTMODE='hotplug'
BOOTPROTO='none'
[connection]
id=bond-slave-ens3
type=ethernet
interface-name=ens3
master=mgmt-bo
slave-type=bond
encoding: ""
ownerstring: ""
```
Expand Down
15 changes: 15 additions & 0 deletions docs/networking/clusternetwork.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,15 @@ During installation of the first cluster node, you can configure the MTU value f

### Add Secondary VLAN Interfaces

:::info DOCS TODO

figure out how to do this. It probably means:
- Add a new connection based on vlan-mgmt
- Add the VLAN id to vlans in bridge-mgmt connection profile
- Add the VLAN id to bridge-port in bond-mgmt connection profile

:::
Copy link
Contributor

@rrajendran17 rrajendran17 Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have closed this harvester/harvester#9221 (comment) issue as it is taken care part of this documentation.

Copy link
Contributor

@rrajendran17 rrajendran17 Oct 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for the secondary interfaces, I believe user might need to add the following contents to the /oem_90_custom.yaml file and reboot the host.(below is an example and vlan id and ip address changes as per requirement of the user) This adds vlan id to the mgmt-br and also creates the secondary vlan interface and assigns ip address to it, if its method is auto, it gets ip address from DHCP.

           - path: /etc/NetworkManager/system-connections/bridge-mgmt.nmconnection
              permissions: 384
              owner: 0
              group: 0
              content: |
                [connection]
                id=bridge-mgmt
                type=bridge
                interface-name=mgmt-br

                [ethernet]


                [bridge]
                forward-delay=0
                stp=false
                vlan-filtering=true
                vlan-default-pvid=1
                vlans=2021

                [ipv4]
                method=disabled



                [ipv6]
                method=disabled
              encoding: ""
              ownerstring: ""
            - path: /etc/NetworkManager/system-connections/vlan-mgmt.nmconnection
              permissions: 384
              owner: 0
              group: 0
              content: |-
                [connection]
                id=vlan-mgmt
                type=vlan

                [ethernet]


                [vlan]
                flags=1
                id=2021
                parent=mgmt-br

                [ipv4]

                method=manual
                address1=10.115.252.74/23,10.115.253.254

                [ipv6]
                method=disabled
              encoding: ""
              ownerstring: ""

To add the vlan to mgmt-bo, user could create a VM Network with vlan id under mgmt cluster network.(an alternate option)


Add the following commands to the `/oem/90_custom.yaml` file and reboot the node.

- `/etc/wicked/scripts/setup_bond.sh` section
Expand Down Expand Up @@ -167,6 +176,12 @@ Exercise extreme caution when editing `/oem/90_custom.yaml`. Do not change other

1. Change the MTU value in the `/oem/90_custom.yaml` file on each node.

:::info DOCS TODO

files are `/etc/NetworkManager/system-connections/{bond-mgmt,bridge-mgmt,vlan-mgmt}.nmconnection`, and you need to add the line `mtu=whatever` to the `[ethernet]` section if it's not already present (it won't be in the default case of 1500)

:::

Locate the following paths, and then change the value in `MTU=1500`.

- `path: /etc/sysconfig/network/ifcfg-mgmt-bo`
Expand Down
2 changes: 1 addition & 1 deletion docs/troubleshooting/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ stages:
...
initramfs:
- commands:
- rm -f /etc/sysconfig/network/ifroute-mgmt-br
- ...
files:
- path: /etc/rancher/rancherd/config.yaml
permissions: 384
Expand Down
10 changes: 10 additions & 0 deletions docs/upgrade/automatic.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,16 @@ Harvester does not support downgrades. This restriction helps prevent unexpected

:::

:::info DOCS TODO

Add information about NetworkManager migration when upgrading from v1.6.x to v1.7.0,
notably that configuration will be generated based on `/oem/harvester.config` and will
be saved in /oem/91_networkmanager.yaml. See https://github.com/harvester/harvester/pull/9039
for more information.

:::


## Upgrade paths

The following table outlines the supported upgrade paths.
Expand Down