Prefer additional network over user network

If using additional network (e.g. lima:shared), override the dhcp rote
metric so the additional metric is preferred.

This fixes issues with k8s components (like submariner) that use the
first default route and break since the user network has not
connectivity between vms. With this change they always use the
additional network.

Example routes with this change when using socket_vmnet:

    $ ip route
    default via 192.168.105.1 dev lima0 proto dhcp src 192.168.105.7 metric 100
    default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
    192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
    192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200
    192.168.105.0/24 dev lima0 proto kernel scope link src 192.168.105.7 metric 100
    192.168.105.1 dev lima0 proto dhcp scope link src 192.168.105.7 metric 100

Example routes without additional network:

    $ ip route
    default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
    192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
    192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200

Another way to solve this issue is to fixup the metric in the
provisioning script as done in RamenDR:
https://github.com/RamenDR/ramen/blob/c02119785e734e15511236edd935c04ff71b6646/test/drenv/providers/lima/k8s.yaml#L37

But I think it is better to fix this in lima, since the current network
configuration is very problematic.

Signed-off-by: Nir Soffer <[email protected]>

Author: nirs

pkg/cidata/cidata.TEMPLATE.d/network-config

@@ -6,6 +6,12 @@ ethernets:
       macaddress: '{{$nw.MACAddress}}'
     dhcp4: true
     set-name: {{$nw.Interface}}
+    dhcp4-overrides:
+    {{- if (eq $nw.Interface $.SlirpNICName) }}
+      route-metric: 200
+    {{- else }}
+      route-metric: 100
+    {{- end }}
     {{- if and (eq $nw.Interface $.SlirpNICName) (gt (len $.DNSAddresses) 0) }}
     nameservers:
       addresses: