DRS failed for cluster if have another cluster in zone

[yashi4engg]

ISSUE TYPE

• Bug Report

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

We have zone1 where we have two clusters ex- cluster01 and cluster02 ,
Where cluster01 have 3 hypervisors with same h/w model and cluster02 has one hypervisor with dfifferent h/w model.
I have enabled drs setiings in globals setting and then disbaled it for cluster02 in cluster setting "drs.automatic.enable -- false" but enabled for cluster01 .

In the above scenario drs plan failed with below logs -

2024-02-08 10:26:33,532 DEBUG [c.c.s.ManagementServerImpl] (VMSchedulerPollTask:ctx-0bba1590) (logid:f0f7966b) Hosts having capacity and suitable for migration: [Host {"id":25,"name":"node-cluster01","type":"Routing","uuid":"5d145861-e4ad-4f94-a805-266711321d59"}, Host {"id":40,"name":"node-cluster02","type":"Routing","uuid":"66b47d2a-b047-452c-b5d3-65c160666b50"}, Host {"id":48,"name":"node-cluster01","type":"Routing","uuid":"da1be373-cd08-4ae1-948f-7592daabb3fc"}]
2024-02-08 10:26:33,535 ERROR [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-0bba1590) (logid:f0f7966b) Unable to generate DRS plans for cluster Cluster-Z01 [id=5366c5fb-0ed0-4caf-b2c7-93ebea15a717]

If i disable host from cluster02 our drs works as expected and migrated VMs based on load . But when we have both clusters and all nodes in cluster enabled its failed to generate plan.

Below are cluster level settings for reference from DB.
621 | 1 | drs.automatic.enable | true |
| 622 | 1 | drs.automatic.interval | 10 |
| 623 | 1 | drs.imbalance | 0.4 |
| 624 | 16 | drs.automatic.enable | false

Even tried by keep it disabled in global settings and then enable it for just one cluster in cluster settings.

As our use case is where we have multiple clusters in same zone with different type of h/w.

STEPS TO REPRODUCE

EXPECTED RESULTS

[rohityadavcloud]

cc @vishesh92

[vishesh92]

@yashi4engg I am not able to reproduce the issue. DRS runs independently for each cluster and this shouldn't be happening.

Can you share the complete logs including the error as well as more details about your clusters?

[yashi4engg]

@vishesh92 -- While trying to reproduce above error. Below is our configs :-

Primary Storage - We are using zone wide primary storages for both clusters.

Cluster 1 have 3 hypervisors and able to run DRS if i disable hosts from cluster02 .

Cluster02 - have 1 hypervisors which is different hardware from hosts in cluster01. DRS is disabled in cluster settings.

[vishesh92]

@yashi4engg The error log you have shared seems to be truncated. Can you share the complete error log?
Also, a cloudstack cluster supports only one hypervisor. It's possible the error is happening because of your current setup. The complete error logs might help in identifying the issue.

[yashi4engg]

We were able to fix above errors and good to close this.

Now DRS is working as expected for us . It was due to zone level storage pool.

[yashi4engg]

@vishesh92 -- it works fine where we have primary storage as NFS ..But its not working where we are using OCFS2 FS as primary storage pool in cluster.Even while generating DRS plan manually i am not seeing error it shows success but not getting any plan .

below are logs -
[root@test01 ~]# cat /var/log/cloudstack/management/management-server.log |grep -i edcf30ae
2024-04-16 10:43:57,568 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) ClusterDRS.poll is being called at 2024-04-16 14:44:00 GMT
2024-04-16 10:43:57,579 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) Removed 0 old drs migration plans
[root@test01 ~]# cat /var/log/cloudstack/management/management-server.log |grep -i edcf30ae
2024-04-16 10:43:57,568 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) ClusterDRS.poll is being called at 2024-04-16 14:44:00 GMT
2024-04-16 10:43:57,579 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-cf16f762) (logid:edcf30ae) Removed 0 old drs migration plans

where as in my cluster VM count is like, i set drs.imbalance to 1.0 as well -
node1 - 150VM
node2 - 150 VM
node3 - 1 VM