SROS (SR-SIM) DUAL CPM mgmt IP race condition

[jcpvdm]

When a SR-SIM CPM container starts up, the following process happens (simplified):

1. python script reads the mgmt IP from linux eth0 interface,
2. bof config is populated with the mgmt IP.
3. sros process is started, and the IP is deleted from linux eth0 interface.

Now... the problem may happen when a node includes two CPM containers (Dual-CPM). Because both CPMs share the same eth0 on the same namespace, if one CPM container reach step 3 before the other CPM finish step 1, then the later CPM won't be able to fetch the mgmt IP address, which results in a broken node.

The issue is more likely to happen on a loaded server, where many containers start concurrently.

[sacckth]

can you try to add a startup-delay to the container? You could also in theory override CPM-B's second mgmt interface not to use eth0.

[jcpvdm]

@sacckth how would you override CPM-B to not to use eth0? Even assuming this is possible, then I think it would be required to join both CPM-A and CPM-B management interfaces in a bridge, and then connect that bridge to the host. (eth0, eth0-A and eth0-B in the same bridge)...

Adding startup-delay to just one of the CPM containers I think would make the problem worse, since that basically would guarantee that one of the CPM containers reach step 3 (and remove IP address from eth0), before the other CPM container reach step 1.

A workaround for the issue is to pass mgmt IP/route information to SR-SIM via explicit environement variables, rather than let SR-SIM automatically discover it from eth0.

topology:
  kinds:
    nokia_srsim:
      license: /opt/nokia/sros/license.txt
      image: nokia_srsim:25.7.R1
  nodes:
    sr-sim1:
      mgmt-ipv4: 10.128.1.111
      kind: nokia_srsim
      type: sr-7
      env:
        NOKIA_SROS_STATIC_ROUTE_1: 0.0.0.0/0@10.128.1.1
        NOKIA_SROS_ADDRESS_IPV4_ACTIVE: "10.128.1.111/24"
      components:
        - slot: A
        - slot: B
        - slot: 1
        - slot: 2

[kaelemc]

there is only one eth0, as a distributed SR-SIM chassis is a single netns.

The eth0 technically always belongs to the netns owner container (which in your sample topo would be slot: 1, as we sort the ordering to ensure either CPMs can be killed without taking away the netns).

I think you are spot on @jcpvdm; probably on the clab side we can set the env vars for the containers so they don't have to discover based on the mgmt interface, as it seems there is a gleaning race condition.

[sacckth]

Yes, we could do that, but there would have to be some special handling because currently env vars are handled in PreDeploy, before we know what the IP of eth0 will be.