Issues on EL9 deployments

[nvazquez]

Able to consistenly reproduce the issue with the mbxt-kvm-el9 template for management server and KVM hosts:

mbx deploy 422-nightly-el9 mbxt-kvm-el9 mbxt-kvm-el9 https://download.cloudstack.org/testing/nightly/2025-10-13/el9/4.22/

(EL8 deployment works fine)

Deployment fails after installing the cloudstack-agent packages on both hosts, with the following error:

Thank you for installing Apache CloudStack Agent 4.22.0.0-SNAPSHOT !

 * Project website:     https://cloudstack.apache.org/
 * Documentation:       https://docs.cloudstack.apache.org/en/4.22.0.0/
 * Release notes:       https://docs.cloudstack.apache.org/en/4.22.0.0/releasenotes/
 * Join mailing lists:  https://cloudstack.apache.org/mailing-lists.html
 * Take the survey:     https://cloudstack.apache.org/survey.html
 * Report issues:       https://github.com/apache/cloudstack/issues/new


  Running scriptlet: lua-libs-5.4.2-4.el9_0.3.x86_64                      61/61 
  Verifying        : copy-jdk-configs-4.0-3.el9.noarch                     1/61 
  Verifying        : java-17-openjdk-1:17.0.16.0.8-2.el9.alma.1.x86_64     2/61 
  Verifying        : java-17-openjdk-headless-1:17.0.16.0.8-2.el9.alma.    3/61 
  Verifying        : javapackages-filesystem-6.4.0-1.el9.noarch            4/61 
  Verifying        : libfontenc-1.1.3-17.el9.x86_64                        5/61 
  Verifying        : libuv-1:1.42.0-2.el9_4.x86_64                         6/61 
  Verifying        : lm_sensors-libs-3.6.0-10.el9.x86_64                   7/61 
  Verifying        : lua-5.4.4-4.el9.x86_64                                8/61 
  Verifying        : lua-posix-35.0-8.el9.x86_64                           9/61 
  Verifying        : mkfontscale-1.2.1-3.el9.x86_64                       10/61 
  Verifying        : pcp-conf-6.3.7-1.el9_6.x86_64                        11/61 
  Verifying        : pcp-libs-6.3.7-1.el9_6.x86_64                        12/61 
  Verifying        : sysstat-12.5.4-9.el9.x86_64                          13/61 
  Verifying        : ttmkfdir-3.0.9-65.el9.x86_64                         14/61 
  Verifying        : tzdata-java-2025b-1.el9.noarch                       15/61 
  Verifying        : xorg-x11-fonts-Type1-7.5-33.el9.noarch               16/61 
  Verifying        : cifs-utils-7.1-2.el9.x86_64                          17/61 
  Verifying        : cryptsetup-2.7.2-3.el9_6.1.x86_64                    18/61 
  Verifying        : jitterentropy-3.6.0-1.el9.x86_64                     19/61 
  Verifying        : libnvme-1.11.1-1.el9.x86_64                          20/61 
  Verifying        : libwbclient-4.21.3-14.el9_6.x86_64                   21/61 
  Verifying        : lksctp-tools-1.0.19-3.el9_4.x86_64                   22/61 
  Verifying        : rng-tools-6.17-1.el9.x86_64                          23/61 
  Verifying        : rsync-3.2.5-3.el9.x86_64                             24/61 
  Verifying        : samba-client-libs-4.21.3-14.el9_6.x86_64             25/61 
  Verifying        : samba-common-4.21.3-14.el9_6.noarch                  26/61 
  Verifying        : samba-common-libs-4.21.3-14.el9_6.x86_64             27/61 
  Verifying        : cloudstack-agent-4.22.0.0-SNAPSHOT.20251013.noarch   28/61 
  Verifying        : cloudstack-common-4.22.0.0-SNAPSHOT.20251013.noarc   29/61 
  Verifying        : openssl-devel-1:3.2.2-6.el9_5.1.x86_64               30/61 
  Verifying        : openssl-devel-1:3.0.1-43.el9_0.x86_64                31/61 
  Verifying        : cryptsetup-libs-2.7.2-3.el9_6.1.x86_64               32/61 
  Verifying        : cryptsetup-libs-2.4.3-5.el9.x86_64                   33/61 
  Verifying        : device-mapper-9:1.02.202-6.el9.x86_64                34/61 
  Verifying        : device-mapper-9:1.02.185-3.el9.x86_64                35/61 
  Verifying        : device-mapper-event-9:1.02.202-6.el9.x86_64          36/61 
  Verifying        : device-mapper-event-9:1.02.185-3.el9.x86_64          37/61 
  Verifying        : device-mapper-event-libs-9:1.02.202-6.el9.x86_64     38/61 
  Verifying        : device-mapper-event-libs-9:1.02.185-3.el9.x86_64     39/61 
  Verifying        : device-mapper-libs-9:1.02.202-6.el9.x86_64           40/61 
  Verifying        : device-mapper-libs-9:1.02.185-3.el9.x86_64           41/61 
  Verifying        : krb5-libs-1.21.1-8.el9_6.x86_64                      42/61 
  Verifying        : krb5-libs-1.19.1-24.el9_1.x86_64                     43/61 
  Verifying        : libldb-4.21.3-14.el9_6.x86_64                        44/61 
  Verifying        : libldb-2.5.2-1.el9.x86_64                            45/61 
  Verifying        : libtalloc-2.4.2-1.el9.x86_64                         46/61 
  Verifying        : libtalloc-2.3.3-1.el9.x86_64                         47/61 
  Verifying        : libtdb-1.4.12-1.el9.x86_64                           48/61 
  Verifying        : libtdb-1.4.6-1.el9.x86_64                            49/61 
  Verifying        : libtevent-0.16.1-1.el9.x86_64                        50/61 
  Verifying        : libtevent-0.12.0-0.el9.x86_64                        51/61 
  Verifying        : lua-libs-5.4.4-4.el9.x86_64                          52/61 
  Verifying        : lua-libs-5.4.2-4.el9_0.3.x86_64                      53/61 
  Verifying        : lvm2-9:2.03.28-6.el9.x86_64                          54/61 
  Verifying        : lvm2-9:2.03.16-3.el9.x86_64                          55/61 
  Verifying        : lvm2-libs-9:2.03.28-6.el9.x86_64                     56/61 
  Verifying        : lvm2-libs-9:2.03.16-3.el9.x86_64                     57/61 
  Verifying        : openssl-1:3.2.2-6.el9_5.1.x86_64                     58/61 
  Verifying        : openssl-1:3.0.1-43.el9_0.x86_64                      59/61 
  Verifying        : openssl-libs-1:3.2.2-6.el9_5.1.x86_64                60/61 
  Verifying        : openssl-libs-1:3.0.1-43.el9_0.x86_64                 61/61 

Upgraded:
  cryptsetup-libs-2.7.2-3.el9_6.1.x86_64                                        
  device-mapper-9:1.02.202-6.el9.x86_64                                         
  device-mapper-event-9:1.02.202-6.el9.x86_64                                   
  device-mapper-event-libs-9:1.02.202-6.el9.x86_64                              
  device-mapper-libs-9:1.02.202-6.el9.x86_64                                    
  krb5-libs-1.21.1-8.el9_6.x86_64                                               
  libldb-4.21.3-14.el9_6.x86_64                                                 
  libtalloc-2.4.2-1.el9.x86_64                                                  
  libtdb-1.4.12-1.el9.x86_64                                                    
  libtevent-0.16.1-1.el9.x86_64                                                 
  lua-libs-5.4.4-4.el9.x86_64                                                   
  lvm2-9:2.03.28-6.el9.x86_64                                                   
  lvm2-libs-9:2.03.28-6.el9.x86_64                                              
  openssl-1:3.2.2-6.el9_5.1.x86_64                                              
  openssl-devel-1:3.2.2-6.el9_5.1.x86_64                                        
  openssl-libs-1:3.2.2-6.el9_5.1.x86_64                                         
Installed:
  cifs-utils-7.1-2.el9.x86_64                                                   
  cloudstack-agent-4.22.0.0-SNAPSHOT.20251013.noarch                            
  cloudstack-common-4.22.0.0-SNAPSHOT.20251013.noarch                           
  copy-jdk-configs-4.0-3.el9.noarch                                             
  cryptsetup-2.7.2-3.el9_6.1.x86_64                                             
  java-17-openjdk-1:17.0.16.0.8-2.el9.alma.1.x86_64                             
  java-17-openjdk-headless-1:17.0.16.0.8-2.el9.alma.1.x86_64                    
  javapackages-filesystem-6.4.0-1.el9.noarch                                    
  jitterentropy-3.6.0-1.el9.x86_64                                              
  libfontenc-1.1.3-17.el9.x86_64                                                
  libnvme-1.11.1-1.el9.x86_64                                                   
  libuv-1:1.42.0-2.el9_4.x86_64                                                 
  libwbclient-4.21.3-14.el9_6.x86_64                                            
  lksctp-tools-1.0.19-3.el9_4.x86_64                                            
  lm_sensors-libs-3.6.0-10.el9.x86_64                                           
  lua-5.4.4-4.el9.x86_64                                                        
  lua-posix-35.0-8.el9.x86_64                                                   
  mkfontscale-1.2.1-3.el9.x86_64                                                
  pcp-conf-6.3.7-1.el9_6.x86_64                                                 
  pcp-libs-6.3.7-1.el9_6.x86_64                                                 
  rng-tools-6.17-1.el9.x86_64                                                   
  rsync-3.2.5-3.el9.x86_64                                                      
  samba-client-libs-4.21.3-14.el9_6.x86_64                                      
  samba-common-4.21.3-14.el9_6.noarch                                           
  samba-common-libs-4.21.3-14.el9_6.x86_64                                      
  sysstat-12.5.4-9.el9.x86_64                                                   
  ttmkfdir-3.0.9-65.el9.x86_64                                                  
  tzdata-java-2025b-1.el9.noarch                                                
  xorg-x11-fonts-Type1-7.5-33.el9.noarch                                        

Complete!
kex_exchange_identification: read: Connection reset by peer

Was able to verify that mbx ssh works for the management server, but not for both hosts:

root@dell-ubuntu:~# mbx ssh qa2-422-nightly-fail-81f934a1-kvm-mgmt1 
MonkeyBox 🐵 v0.5-pre
Last login: Mon Jan 23 02:11:42 2023 from 172.20.0.1
[root@qa2-422-nightly-fail-81f934a1-kvm-mgmt1 ~]# exit
logout
root@dell-ubuntu:~# mbx ssh qa2-422-nightly-fail-81f934a1-kvm-host1
MonkeyBox 🐵 v0.5-pre
kex_exchange_identification: read: Connection reset by peer

[rohityadavcloud]

Thanks Nicolas. This is a known issue, on my list to rebuild the el8/9/10 templates using Oracle Linux, this is just Alma linux failing its users.

[nvazquez]

Ahh thanks @rohityadavcloud - was not aware it was a known issue, logged it as wasn't able to work it around after some fix attempts. Will close the ticket

[rohityadavcloud]

No worries, I’ll reopen this to serve as reminder for us.