[DNM] Run validate-network service one stage later

[amartyasinha]

Due to some race condition, DNS service is not returning FQDN for compute nodes when validate-network pods kick in. This patch aims to start validate-network service in later stage with hope this random issue gets fixed.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[evallesp]

LGTM!

[openshift-ci]

New changes are detected. LGTM label has been removed.

[fultonj]

-1

If we really think validate-network should be run later, then we should change it in EDPM itself and update our documentation.

@slagle what do you think?

[slagle]

-1

If we really think validate-network should be run later, then we should change it in EDPM itself and update our documentation.

@slagle what do you think?

We don't have a root cause of the issue that indicates such a change is needed. I agree that changing the service order here isn't right either.

[amartyasinha]

If we really think validate-network should be run later, then we should change it in EDPM itself and update our documentation.

We don't have a root cause of the issue that indicates such a change is needed. I agree that changing the service order here isn't right either.

@evallesp WDYT about this? It is true that we don't have a root cause. This issue is happening randomly and we are trying to avoid it by changing the order of services. What should be the next step as reordering services without root cause is discouraged.

[amartyasinha]

@fultonj @slagle I agree with you, so I looked into the playbook, which is called through the configure-network service.

The playbook led me to this edpm_network_config role, and it seems to be the last stage in the configure-network service. IIUC, the role is doing all the configuration, but the loading of dnsmasq is still in progress. Then, the validate-network service is initiated, and sometimes dnsmasq is not ready, causing the issue.

@sdatko Maybe this should be handled by networking experts rather than the ci-framework folks.

@karelyatin Maybe you could help?

[slagle]

@fultonj @slagle I agree with you, so I looked into the playbook, which is called through the configure-network service.

The playbook led me to this edpm_network_config role, and it seems to be the last stage in the configure-network service. IIUC, the role is doing all the configuration, but the loading of dnsmasq is still in progress. Then, the validate-network service is initiated, and sometimes dnsmasq is not ready, causing the issue.

We need to know why dnsmasq isn't ready. It's deployed as part of the OpenStackControlPlane resource, so when that reports ready, dnsmasq should be ready as well.

Are we waiting for OpenStackControlPlane to be Ready=True before starting the dataplane deployment? We document this as a requirement in the docs, but don't show an example of using the oc wait command to check, which we should probably add. The command is what is done by the openstack_wait_deploy target in install_yamls.

[karelyatin]

<< @karelyatin Maybe you could help?

Do we have logs of job where it failed?

[amartyasinha]

Unfortunately, logs have been rotated. And since it's occurring rarely, it is difficult to reproduce it.

…

On Wed, Aug 13, 2025, 18:18 Yatin Karel ***@***.***> wrote: *karelyatin* left a comment (openstack-k8s-operators/architecture#595) <#595 (comment)> << @karelyatin <https://github.com/karelyatin> Maybe you could help? Do we have logs of job where it failed? — Reply to this email directly, view it on GitHub <#595 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATLBCGQYIAEMFXVMUEL6TYD3NMXX3AVCNFSM6AAAAACDBA6NCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCOBTG43TSOJQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[karelyatin]

<< Unfortunately, logs have been rotated. And since it's occurring rarely, it
is difficult to reproduce it.
ack then let's hold this and edpm-ansible PR as slagle pointed out we should check the issue once we have logs and/or reproducer

[evallesp]

recheck

[fultonj]

James wrote:

Are we waiting for OpenStackControlPlane to be Ready=True before starting the dataplane deployment?

We do in HCI for example:

https://github.com/openstack-k8s-operators/architecture/blob/main/automation/vars/hci.yaml#L35

I see a similar call in these jobs:

[zuul@controller-0 vars]$ grep "openstackcontrolplane" *.yaml
bgp-l3-xl.yaml:            oc -n openstack wait openstackcontrolplane
bgp_dt01.yaml:            oc -n openstack wait openstackcontrolplane
bmo01.yaml:            oc -n openstack wait openstackcontrolplane
dz-storage.yaml:            oc -n openstack wait openstackcontrolplane
osasinfra-ipv6.yaml:            oc -n openstack wait openstackcontrolplane
osasinfra.yaml:            oc -n openstack wait openstackcontrolplane
uni01alpha.yaml:            oc -n openstack wait openstackcontrolplane
uni02beta.yaml:            oc -n openstack wait openstackcontrolplane
uni05epsilon.yaml:            oc -n openstack wait openstackcontrolplane
uni06zeta.yaml:            oc -n openstack wait openstackcontrolplane
uni07eta.yaml:            oc -n openstack wait openstackcontrolplane
[zuul@controller-0 vars]$

[zuul@controller-0 vars]$ grep controlplane *.yaml
bgp-l3-xl.yaml:            oc -n openstack wait openstackcontrolplane
bgp-l3-xl.yaml:            controlplane
bgp-l3-xl.yaml:          - name: Create BGPConfiguration after controlplane is deployed
bgp_dt01.yaml:            oc -n openstack wait openstackcontrolplane
bgp_dt01.yaml:            controlplane
bmo01.yaml:            oc -n openstack wait openstackcontrolplane
bmo01.yaml:            controlplane
dcn.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
dz-storage.yaml:            oc -n openstack wait openstackcontrolplane
dz-storage.yaml:            controlplane
dz-storage.yaml:          - name: Create BGPConfiguration after controlplane is deployed
hci.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
multi-namespace.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
multi-namespace.yaml:            oc -n openstack2 wait osctlplane controlplane --for condition=Ready
nfv-ovs-dpdk-sriov-hci.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
nova-three-cells.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
nova01alpha.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
nvidia-mdev.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
osasinfra-ipv6.yaml:            oc -n openstack wait openstackcontrolplane
osasinfra-ipv6.yaml:            controlplane
osasinfra.yaml:            oc -n openstack wait openstackcontrolplane
osasinfra.yaml:            controlplane
ovs-dpdk-sriov-2nodesets.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
ovs-dpdk-sriov-networker.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
ovs-dpdk-sriov.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
ovs-dpdk.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
pidone.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
sriov.yaml:            oc -n openstack wait osctlplane controlplane --for condition=Ready
uni01alpha.yaml:            oc -n openstack wait openstackcontrolplane
uni01alpha.yaml:            controlplane
uni02beta.yaml:            oc -n openstack wait openstackcontrolplane
uni02beta.yaml:            controlplane
uni04delta-ipv6.yaml:            oc -n openstack wait osctlplane controlplane
uni04delta.yaml:            oc -n openstack wait osctlplane controlplane
uni05epsilon.yaml:            oc -n openstack wait openstackcontrolplane
uni05epsilon.yaml:            controlplane
uni06zeta.yaml:            oc -n openstack wait openstackcontrolplane
uni06zeta.yaml:            controlplane
uni07eta.yaml:            oc -n openstack wait openstackcontrolplane
uni07eta.yaml:            controlplane
[zuul@controller-0 vars]$ 

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amartyasinha
Once this PR has been reviewed and has the lgtm label, please assign fultonj for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment