define desired behaviour

Author: sjpb

docs/networks.md

@@ -8,11 +8,13 @@ subnets or associated infrastructure such as routers. The requirements are that:
 4. At least one network on each node provides outbound internet access (either
 directly, or via a proxy).
 
-Futhermore, it is recommended that the deploy host has an interface on the
-access network. While it is possible to e.g. use a floating IP on a login node
-as an SSH proxy to access the other nodes, this can create problems in recovering
-the cluster if the login node is unavailable and can make Ansible problems harder
-to debug.
+Addresses on the "access network" are used for `ansible_host` and `k3s` node IPs.
+
+It is recommended that the deploy host either has a direct connection to the
+"access network" or jumps through a host on it which is not part of the appliance.
+Using e.g. a floating IP on a login node as a jumphost creates problems in
+recovering the cluster if the login node is unavailable and can make Ansible
+problems harder to debug.
 
 > [!WARNING]
 > If home directories are on a shared filesystem with no authentication (such
@@ -29,8 +31,8 @@ the OpenTofu variables. These will normally be set in
 need to be overriden for specific environments, this can be done via an OpenTofu
 module as discussed [here](./production.md).
 
-Note that if an OpenStack subnet has a gateway IP defined then nodes with ports
-attached to that subnet will get a default route set via that gateway.
+Note that if an OpenStack subnet has a gateway IP defined then by default nodes
+with ports attached to that subnet get a default route set via that gateway.
 
 ## Single network
 This is the simplest possible configuration. A single network and subnet is
@@ -77,8 +79,9 @@ vnic_types = {
 ## Additional networks on some nodes
 
 This example shows how to modify variables for specific node groups. In this
-case a baremetal node group has a second network attached. As above, only a
-single subnet can have a gateway IP.
+case a baremetal node group has a second network attached. Here "subnetA" must
+have a gateway IP defined and "subnetB" must not, to avoid routing problems on
+the multi-homeed compute nodes.
 
 ```terraform
 cluster_networks = [
@@ -109,3 +112,76 @@ compute = {
 }
 ...
 ```
+
+## Multiple networks with non-default gateways
+
+In some multiple network configurations it may be necessary to manage default
+routes rather than them being automatically created from a subnet gateway.
+This can be done using the tofu variable `gateway_ip` which can be set for the
+cluster and/or overriden on the compute and login groups. If this is set:
+- a default route via that address will be created on the appropriate interface
+  during boot if it does not exist
+- any other default routes will be removed
+
+For example the cluster configuration below has a "campus" network with a
+default gateway which provides inbound SSH / ondemand access and outbound
+internet  attached only to the login nodes, and a "data" network attached to
+all nodes. The "data" network has no gateway IP set on its subnet to avoid dual
+default routes and routing conflicts on the multi-homed login nodes, but does
+have outbound connectivity via a router:
+
+```terraform
+cluster_networks = [
+  {
+    network = "data" # access network, CIDR 172.16.0.0/23
+    subnet = "data_subnet"
+  }
+]
+
+login = {
+  interactive = {
+    nodes = ["login-0"]
+    extra_networks = [
+      {
+        network = "campus"
+        subnet = "campus_subnet"
+      }
+    ]
+  }
+}
+compute = {
+  general = {
+    nodes = ["compute-0", "compute-1"]
+  }
+  gateway_ip =  "172.16.0.1" # Router interface
+}
+```
+
+If there is no default route at all (either from a subnet gateway or from
+`gateway_ip`) then a dummy route is created via the access network interface to
+ensure [correct](https://docs.k3s.io/installation/airgap#default-network-route)
+`k3s` operation.
+
+## Proxies
+
+If some nodes have no outbound connectivity via any networks, the cluster can
+be configured to deploy a [squid proxy](https://www.squid-cache.org/) on a node
+with outbound connectivity. Assuming the `compute` and `control` nodes have no
+outbound connectivity and the `login` node does, the minimal configuration for
+this is:
+
+```yaml
+# environments/$SITE/inventory/groups:
+[squid:children]
+login
+[proxy:children]
+control
+compute
+```
+
+```yaml
+# environments/$SITE/inventory/group_vars/all/squid.yml:
+# these are just examples
+squid_cache_disk: 1024 # MB
+squid_cache_mem: '12 GB'
+```