diff --git a/docs/install/harvester-configuration.md b/docs/install/harvester-configuration.md index 13c033d6f4..a0431c28fb 100644 --- a/docs/install/harvester-configuration.md +++ b/docs/install/harvester-configuration.md @@ -266,10 +266,10 @@ DNS resolution is unavailable in the `after-install-chroot stage`, and the `name ```yaml os: after_install_chroot_commands: - - "rm -f /etc/resolv.conf && echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf" + - "echo 'nameserver 8.8.8.8' | sudo tee /etc/resolv.conf" - "mkdir /usr/local/bin" - "curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh" - - "rm -f /etc/resolv.conf && ln -s /var/run/netconfig/resolv.conf /etc/resolv.conf" + - "rm -f /etc/resolv.conf" ``` diff --git a/docs/install/requirements.md b/docs/install/requirements.md index 1215405801..f14b45040f 100644 --- a/docs/install/requirements.md +++ b/docs/install/requirements.md @@ -88,7 +88,7 @@ Harvester nodes require the following port connections or inbound rules. Typical | TCP | 9796 | Harvester management and compute nodes | Node-exporter | | TCP | 30000-32767 | Harvester management and compute nodes | NodePort port range | | TCP | 22 | Harvester management and compute nodes | sshd | -| UDP | 68 | Harvester management and compute nodes | Wicked | +| UDP | 68 | Harvester management and compute nodes | NetworkManager | | TCP | 3260 | Harvester management and compute nodes | iscsid | ### Port Requirements for Integrating Harvester with Rancher diff --git a/docs/install/update-harvester-configuration.md b/docs/install/update-harvester-configuration.md index 6f735ec519..ca19e5a933 100644 --- a/docs/install/update-harvester-configuration.md +++ b/docs/install/update-harvester-configuration.md @@ -20,6 +20,8 @@ This page describes how to edit some of the most-requested Harvester configurati If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples will be `/oem/99_custom.yaml`. Please substitute the value if needed. +If you upgrade from a version before `v1.7.0`, any changes to network and DNS configuration need to be made in the `/oem/91_networkmanager.yaml` file. Please substitute the value if needed. + ::: ## DNS servers @@ -27,16 +29,16 @@ If you upgrade from a version before `v1.1.2`, the `cloud-init` file in examples ### Runtime change 1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details. -1. Edit `/etc/sysconfig/network/config` and update the following line. Use a space to separate DNS server addresses if there are multiple servers. +1. If the management interface _is not_ configured to use a VLAN, run the following command: ``` - NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1" + nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br ``` -1. Update and reload the configuration with the following command: +1. If the management interface _is_ configured to use a VLAN, run the following commands. Replace `VLAN_ID` with the actal ID of the VLAN. If in doubt, run `nmcli con` to see the configured connections and devices. ``` - netconfig update + nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.VLAN_ID ``` 1. Confirm the file `/etc/resolv.conf` contains the correct DNS servers with the `cat` command: @@ -59,24 +61,35 @@ Beginning with v1.1.2, the persistent name of the cloud-init file is `/oem/90_cu When upgrading from an earlier version to `v1.1.2` or later, Harvester retains the old file name (`/oem/99_custom.yaml`) to avoid confusion. You can manually rename the file to `/oem/90_custom.yaml` if necessary. +If you have upgraded from a version before `v1.7.0`, the changes below need to be made in the `/oem/91_networkmanager.yaml` file. + 1. Backup the elemental `cloud-init` file `/oem/90_custom.yaml` as follows: ``` cp /oem/90_custom.yaml /oem/install/90_custom.yaml.$(date --iso-8601=minutes) ``` -1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.initramfs[0].commands`. The `commands` array must contain a line to manipulate the `NETCONFIG_DNS_STATIC_SERVERS` config. Add the line if the line doesn't exist. +1. Edit `/oem/90_custom.yaml` and update the value under the yaml path `stages.network[0].commands`. The `commands` array must contain a line to manipulate the NetworkManager DNS configuration for the management interface. This is the exact same command used above when making the change at runtime. Add the line if the line doesn't exist. - The following example adds a line to change the `NETCONFIG_DNS_STATIC_SERVERS` config: + The following example adds a line to configure DNS servers when not using a VLAN on the management interface: ``` stages: - initramfs: + network: + - commands: + - nmcli con modify bridge-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br + ``` + + The following example adds a line to configure DNS servers when using VLAN 2017 on the management interface: + + ``` + stages: + network: - commands: - - sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="8.8.8.8 1.1.1.1"/' /etc/sysconfig/network/config + - nmcli con modify vlan-mgmt ipv4.dns 8.8.8.8,1.1.1.1 && nmcli device reapply mgmt-br.2017 ``` - Replace the DNS server addresses and save the file. Harvester sets up new servers after rebooting. + Replace the DNS server addresses and VLAN ID if applicable and save the file. Harvester sets up new servers after rebooting. ## NTP servers @@ -109,6 +122,21 @@ You can update the slave interfaces of Harvester's management bonding interface ### Runtime change +:::info DOCS TODO + +Procedure is to add a new NM connection profile for the new interface, e.g.: `/etc/NetworkManager/system-connections`, with content: +``` +[connection] +id=bond-slave-eno50 +type=ethernet +interface-name=eno50 +master=mgmt-bo +slave-type=bond +``` +...then `nmcli con reload` and probably `nmcli device reapply mgmt-bo`. + +::: + 1. Log in to a Harvester node and become root. See [how to log into a Harvester node](../troubleshooting/os.md#how-to-log-in-to-a-harvester-node) for more details. 1. Identify the interface names with the following command: @@ -138,6 +166,11 @@ You can update the slave interfaces of Harvester's management bonding interface ### Configuration persistence +:::info DOCS TODO + +Files as in above TODO, and note `/oem/91_networkmanager.yaml` for pre v1.7.0 systems + +::: 1. Backup the elemental cloud-init file `/oem/90_custom.yaml` as follows: diff --git a/docs/logging/harvester-logging.md b/docs/logging/harvester-logging.md index 9a81f51360..3276d1f7d4 100644 --- a/docs/logging/harvester-logging.md +++ b/docs/logging/harvester-logging.md @@ -47,6 +47,12 @@ In Harvester's practice, the `Logging`, `Audit` and `Event` shares one architect The Harvester logging infrastructure allows you to aggregate Harvester logs into an external service such as [Graylog](https://www.graylog.org), [Elasticsearch](https://www.elastic.co/elasticsearch/), [Splunk](https://www.splunk.com/), [Grafana Loki](https://grafana.com/oss/loki/) and others. ### Collected Logs +:::info DOCS TODO + +Confirm this does NetworkManager logs and fix it if not. + +::: + See below for a list logs that are collected: - Logs from all cluster `Pods` - Kernel logs from each `node` diff --git a/docs/networking/best-practice.md b/docs/networking/best-practice.md index 538f2acf1b..e21ec33604 100644 --- a/docs/networking/best-practice.md +++ b/docs/networking/best-practice.md @@ -45,28 +45,40 @@ Before any action is taken, it is important to collect the current network infor Example: A NIC named `ens3` was added to the `mgmt-bo` bond interface. ``` - - path: /etc/sysconfig/network/ifcfg-mgmt-bo + - path: /etc/NetworkManager/system-connections/bond-mgmt.nmconnection permissions: 384 owner: 0 group: 0 - content: |+ - STARTMODE='onboot' - BONDING_MASTER='yes' - BOOTPROTO='none' - POST_UP_SCRIPT="wicked:setup_bond.sh" - BONDING_SLAVE_0='ens3' - BONDING_MODULE_OPTS='miimon=100 mode=active-backup ' - DHCLIENT_SET_DEFAULT_ROUTE='no' + content: |- + [connection] + id=bond-mgmt + type=bond + interface-name=mgmt-bo + master=mgmt-br + slave-type=bridge + + [ethernet] + + [bond] + miimon=100 + mode=active-backup + + [bridge-port] + vlans=1 pvid untagged encoding: "" ownerstring: "" - - path: /etc/sysconfig/network/ifcfg-ens3 + - path: /etc/NetworkManager/system-connections/bond-slave-ens3.nmconnection permissions: 384 owner: 0 group: 0 content: | - STARTMODE='hotplug' - BOOTPROTO='none' + [connection] + id=bond-slave-ens3 + type=ethernet + interface-name=ens3 + master=mgmt-bo + slave-type=bond encoding: "" ownerstring: "" ``` diff --git a/docs/networking/clusternetwork.md b/docs/networking/clusternetwork.md index d995d3b069..3e3a83177e 100644 --- a/docs/networking/clusternetwork.md +++ b/docs/networking/clusternetwork.md @@ -102,6 +102,15 @@ During installation of the first cluster node, you can configure the MTU value f ### Add Secondary VLAN Interfaces +:::info DOCS TODO + +figure out how to do this. It probably means: +- Add a new connection based on vlan-mgmt +- Add the VLAN id to vlans in bridge-mgmt connection profile +- Add the VLAN id to bridge-port in bond-mgmt connection profile + +::: + Add the following commands to the `/oem/90_custom.yaml` file and reboot the node. - `/etc/wicked/scripts/setup_bond.sh` section @@ -167,6 +176,12 @@ Exercise extreme caution when editing `/oem/90_custom.yaml`. Do not change other 1. Change the MTU value in the `/oem/90_custom.yaml` file on each node. + :::info DOCS TODO + + files are `/etc/NetworkManager/system-connections/{bond-mgmt,bridge-mgmt,vlan-mgmt}.nmconnection`, and you need to add the line `mtu=whatever` to the `[ethernet]` section if it's not already present (it won't be in the default case of 1500) + + ::: + Locate the following paths, and then change the value in `MTU=1500`. - `path: /etc/sysconfig/network/ifcfg-mgmt-bo` diff --git a/docs/troubleshooting/installation.md b/docs/troubleshooting/installation.md index 5393544362..b39c2fb31a 100644 --- a/docs/troubleshooting/installation.md +++ b/docs/troubleshooting/installation.md @@ -70,7 +70,7 @@ stages: ... initramfs: - commands: - - rm -f /etc/sysconfig/network/ifroute-mgmt-br + - ... files: - path: /etc/rancher/rancherd/config.yaml permissions: 384 diff --git a/docs/upgrade/automatic.md b/docs/upgrade/automatic.md index 2d9ca2d4af..e62b21b461 100644 --- a/docs/upgrade/automatic.md +++ b/docs/upgrade/automatic.md @@ -28,6 +28,16 @@ Harvester does not support downgrades. This restriction helps prevent unexpected ::: +:::info DOCS TODO + +Add information about NetworkManager migration when upgrading from v1.6.x to v1.7.0, +notably that configuration will be generated based on `/oem/harvester.config` and will +be saved in /oem/91_networkmanager.yaml. See https://github.com/harvester/harvester/pull/9039 +for more information. + +::: + + ## Upgrade paths The following table outlines the supported upgrade paths.