ESI: fully move MOC-R4PCC02U37,MOC-R4PCC02U38 into ESI under a new nerc-ocp-test project

[jtriley]

Motivation

There are 2xH100 hosts, MOC-R4PCC02U37 and MOC-R4PCC02U38, that were moved to ESI OBM network but setup manually network-wise and provisioned into the nerc-ocp-test cluster for the NERC project. We'd like to fully move these hosts into a new project called nerc-ocp-test so we can test setting up RDMA within ocp-test via ESI as would be done in production for NAIRR projects.

Completion Criteria

The 2xH100 hosts (MOC-R4PCC02U37, MOC-R4PCC02U38) are leased to new nerc-ocp-test project in ESI.

Description

• Create nerc-ocp-test project with all members from nerc-admins ESI project added as members/admins of the nerc-ocp-test project
• Add nerc-openshift-test-frontend and nerc-openshift-test-backend networks to nerc-ocp-test project
• Move MOC-R4PCC02U37, MOC-R4PCC02U38 fully into ESI management
• Lease MOC-R4PCC02U37, MOC-R4PCC02U38 to nerc-ocp-test

Completion dates

Desired - 2026-01-15
Required - 2026-01-16

[tzumainn]

I've created nerc-ocp-test and populated its members. I'll work on the nodes once they're out from ocp-test

[jtriley]

I've created nerc-ocp-test and populated its members. I'll work on the nodes once they're out from ocp-test

All set.

[tzumainn]

Networks should be shared with nerc-ocp-test now!

[tzumainn]

nodes have inspected/cleaned successfully, and are now assigned to nerc-ocp-test - let me know if there are any issues!

[jtriley]

nodes have inspected/cleaned successfully, and are now assigned to nerc-ocp-test - let me know if there are any issues!

Thanks @tzumainn! Things are mostly working OK as far as setting up trunk ports and getting them added back in to the ocp-test cluster. I've been able to successfully do that.

This may be another issue altogether but I'm hitting some strange behavior when attaching mac addresses to a simple ESI-native/local network:

$ openstack esi node network attach --network rdma-network --mac-address a0:88:c2:27:c4:78 MOC-R4PCC02U38
Expecting value: line 1 column 1 (char 0)

I got this command to work for other NICs on the same host but then it looks like it's assigning duplicate static IP? Not a big deal given that we've disabled DHCP and gateway on the subnet but still seems like something went sideways:

| MOC-R4PCC02U38 | a0:88:c2:27:c4:70 | esi-MOC-R4PCC02U38-rdma-network                                     | rdma-network (456)                  | 192.168.216.31  | None             | None        |
| MOC-R4PCC02U38 | a0:88:c2:27:c4:74 | esi-MOC-R4PCC02U38-rdma-network                                     | rdma-network (456)                  | 192.168.216.31  | None             | None        |
| MOC-R4PCC02U38 | a0:88:c2:27:c4:78 | esi-MOC-R4PCC02U38-rdma-network                                     | rdma-network (456)                  | 192.168.216.31  | None             | None        |
| MOC-R4PCC02U38 | a0:88:c2:27:c4:7c | None                                                                | None                                | None            | None             | None        |
| MOC-R4PCC02U38 | 38:7c:76:94:97:38 | esi-ocp-test-MOC-R4PCC02U38-nerc-openshift-test-frontend-trunk-port | nerc-openshift-test-frontend (2174) |                 | None             | None        |

The ESI node network attach command seems to always use the 192.168.216.31 address for this host for some reason.
However, the openstack port list command shows a different situation:

$ openstack port list | grep -i rdma
| 4a200337-e8fa-4f39-9d60-fd424d9b2652 | esi-MOC-R4PCC02U38-rdma-network                                     | a0:88:c2:27:c4:78 | ip_address='192.168.216.31', subnet_id='4852cce7-86fb-4aa8-9d3c-76ff291cf7fd'  | DOWN   |
| 54bc7a39-aee9-4e7b-aa89-214791fc7bc7 | esi-MOC-R4PCC02U37-rdma-network                                     | a0:88:c2:27:c3:6c | ip_address='192.168.216.148', subnet_id='4852cce7-86fb-4aa8-9d3c-76ff291cf7fd' | ACTIVE |
| 8bcff08f-0823-4591-a33c-212343bf5646 | esi-MOC-R4PCC02U37-rdma-network                                     | a0:88:c2:27:c3:64 | ip_address='192.168.216.183', subnet_id='4852cce7-86fb-4aa8-9d3c-76ff291cf7fd' | ACTIVE |
| 983d3ecf-f06d-46d8-8298-641e9ea77d11 | esi-MOC-R4PCC02U37-rdma-network                                     | a0:88:c2:27:c3:60 | ip_address='192.168.216.246', subnet_id='4852cce7-86fb-4aa8-9d3c-76ff291cf7fd' | ACTIVE |
| fb7353f9-a2c1-4102-8330-19b5c2e466e2 | esi-MOC-R4PCC02U37-rdma-network                                     | a0:88:c2:27:c3:68 | ip_address='192.168.216.43', subnet_id='4852cce7-86fb-4aa8-9d3c-76ff291cf7fd'  | ACTIVE |
[9:05](https://massopencloud.slack.com/archives/D05LQEY1ZGQ/p1768529138548109)

Also worth noting that the duplicated IP shown by esi node network command (192.168.216.31) is shown as DOWN

[tzumainn]

Interesting - I created ports manually and attached them to MOC-R4PCC02U38, but they all seem to be DOWN still. @hakasapl is there something that might be causing this on the switch?

[hakasapl]

@tzumainn just catching up on this apologies for the delay. Did we resolve this by deleting the bonds already?

[tzumainn]

@hakasapl oh, I thought you deleted the bonds (I'm not sure how to do that)

[hakasapl]

@tzumainn I did delete them I was just verifying that has solved this issue?

[tzumainn]

Oh, got it! I believe @jtriley was able to use the nodes after the fix - Justin, can you confirm?