Certificate error when downloading ISO (keystore not set up correctly)

[salfers]

ISSUE TYPE

• Bug Report

COMPONENT NAME

System VM?

CLOUDSTACK VERSION

CloudStack 4.19.0.1
14a42b786e681e2dd52fdd43b4a30c3d8fac96a7

CONFIGURATION

Nothing relevant as far as I can see.

OS / ENVIRONMENT

Alma Linux 9.3 x86_64

SUMMARY

Adding ISO files from HTTPS URLs does not work and the UI shows a certificate error.

There have apparently been similar issues in the past (#2541) that were patched manually but my guess here is that the Java certificate store is not set up correctly.

STEPS TO REPRODUCE

Go to Images > ISOs and create a new ISO with URL https://download.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-NET-x86_64-Media.iso

EXPECTED RESULTS

ISO is correctly downloaded

ACTUAL RESULTS

Shows a DOWNLOAD_ERROR and this text:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[shwstppr]

@salfers can you please check if adding ISO works if you select Direct Download?

[shwstppr]

@salfers I just tested adding ISO with the given link and it worked fine for me (ignore the OS. I didn't change that),

In my test env, I was using CentOS 7.6 for both hypervisor and management server. But since download of ISO for non-directdownload ISO is carried out by SSVM so I'm not sure if that should affect the behaviour

[salfers]

@salfers can you please check if adding ISO works if you select Direct Download?

Adding it works, as soon as I start a VM it fails with

Unable to orchestrate start VM instance {"id":324,"instanceName":"i-2-324-VM","type":"User","uuid":"c4542c3d-6df7-4016-9354-f8248e956efa"} due to [Template 230 could not be downloaded on pool 2, failing after trying on several hosts Details: Unable to download template: Could not find volume 3c658b8c-4da7-496c-9741-f40de8bd5774.iso: Storage volume not found: no storage vol with matching name '3c658b8c-4da7-496c-9741-f40de8bd5774.iso'].

management-server.log shows:

2024-05-22 09:38:12,787 WARN  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-8a0dd68f job-3564/job-3566 ctx-7c4e0d7c) (logid:51002c3d) Unable to orchestrate start VM instance {"id":324,"instanceName":"i-2-324-VM","type":"User","uuid":"c4542c3d-6df7-4016-9354-f8248e956efa"} due to [Template 230 could not be downloaded on pool 2, failing after trying on several hosts Details: Unable to download template: Could not find volume 3c658b8c-4da7-496c-9741-f40de8bd5774.iso: Storage volume not found: no storage vol with matching name '3c658b8c-4da7-496c-9741-f40de8bd5774.iso'].
com.cloud.utils.exception.CloudRuntimeException: Template 230 could not be downloaded on pool 2, failing after trying on several hosts Details: Unable to download template: Could not find volume 3c658b8c-4da7-496c-9741-f40de8bd5774.iso: Storage volume not found: no storage vol with matching name '3c658b8c-4da7-496c-9741-f40de8bd5774.iso'
        at org.apache.cloudstack.direct.download.DirectDownloadManagerImpl.sendDirectDownloadCommand(DirectDownloadManagerImpl.java:376)
        at org.apache.cloudstack.direct.download.DirectDownloadManagerImpl.downloadTemplate(DirectDownloadManagerImpl.java:287)
        at org.apache.cloudstack.storage.image.TemplateDataFactoryImpl.getReadyBypassedTemplateOnPrimaryStore(TemplateDataFactoryImpl.java:259)
        at com.cloud.template.TemplateManagerImpl.prepareIso(TemplateManagerImpl.java:1242)
        at com.cloud.template.TemplateManagerImpl.prepareIsoForVmProfile(TemplateManagerImpl.java:630)
        at jdk.internal.reflect.GeneratedMethodAccessor846.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[...]

agent.log on the relevant host:

2024-05-22 09:38:07,807 INFO  [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:null) (logid:51002c3d) The server did not provide the template size, assuming there is enough space to download it
2024-05-22 09:38:07,807 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:51002c3d) Trying to fetch storage pool edf93483-83f6-326f-90be-74077be877a7 from libvirt
2024-05-22 09:38:08,178 INFO  [direct.download.DirectTemplateDownloaderImpl] (agentRequest-Handler-5:null) (logid:51002c3d) Downloading template 230 from https://download.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-NET-x86_64-Media.iso to: /var/lib/libvirt/images/template/2/230/openSUSE-Leap-15.5-NET-x86_64-Media.iso
2024-05-22 09:38:10,454 INFO  [direct.download.DirectTemplateDownloaderImpl] (agentRequest-Handler-5:null) (logid:51002c3d) No checksum provided, skipping checksum validation
2024-05-22 09:38:10,455 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:51002c3d) Trying to fetch storage pool edf93483-83f6-326f-90be-74077be877a7 from libvirt
2024-05-22 09:38:10,646 WARN  [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-5:null) (logid:51002c3d) Error downloading template 230 due to: Could not find volume 01808b2e-f09e-427a-8d72-8fce630c1fbf.iso: Storage volume not found: no storage vol with matching name '01808b2e-f09e-427a-8d72-8fce630c1fbf.iso'
2024-05-22 09:38:10,673 INFO  [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-3:null) (logid:51002c3d) The server did not provide the template size, assuming there is enough space to download it
2024-05-22 09:38:10,674 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:51002c3d) Trying to fetch storage pool edf93483-83f6-326f-90be-74077be877a7 from libvirt
2024-05-22 09:38:10,827 INFO  [direct.download.DirectTemplateDownloaderImpl] (agentRequest-Handler-3:null) (logid:51002c3d) Downloading template 230 from https://download.opensuse.org/distribution/leap/15.5/iso/openSUSE-Leap-15.5-NET-x86_64-Media.iso to: /var/lib/libvirt/images/template/2/230/openSUSE-Leap-15.5-NET-x86_64-Media.iso
2024-05-22 09:38:12,315 INFO  [direct.download.DirectTemplateDownloaderImpl] (agentRequest-Handler-3:null) (logid:51002c3d) No checksum provided, skipping checksum validation
2024-05-22 09:38:12,315 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:51002c3d) Trying to fetch storage pool edf93483-83f6-326f-90be-74077be877a7 from libvirt
2024-05-22 09:38:12,498 WARN  [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-3:null) (logid:51002c3d) Error downloading template 230 due to: Could not find volume 3c658b8c-4da7-496c-9741-f40de8bd5774.iso: Storage volume not found: no storage vol with matching name '3c658b8c-4da7-496c-9741-f40de8bd5774.iso'
2024-05-22 09:38:27,152 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:3902c3cc) Trying to fetch storage pool 417aa284-c039-491d-926e-89117ddc75c3 from libvirt

[salfers]

I did some debugging on the SSVM and the cloud agent runs with -Djavax.net.ssl.trustStore=./certs/realhostip.keystore.

These are the certificates inside the trust store:

root@s-253-VM:/usr/local/cloud/systemvm# keytool -list -keystore certs/realhostip.keystore 
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 4 entries

cross, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): 18:F8:A7:A1:51:B4:EC:28:08:98:09:3D:F5:BD:53:7C:A0:99:CC:27:74:05:D0:28:1D:E0:DA:DF:D1:44:20:DA
intermed, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): 09:ED:6E:99:1F:C3:27:3D:8F:EA:31:7D:33:9C:02:04:18:61:97:35:49:CF:A6:E1:55:8F:41:1F:11:21:1A:A3
realhostip, Feb 3, 2012, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): ED:76:FC:C1:8A:84:B6:C6:AC:88:FE:9C:F0:22:B2:9D:83:54:7E:5B:5C:92:0B:10:B7:D5:70:27:92:1C:7B:EF
root, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): C3:84:6B:F2:4B:9E:93:CA:64:27:4C:0E:C6:7C:1E:CC:5E:02:4F:FC:AC:D2:D7:40:19:35:0E:81:FE:54:6A:E4

None of the normal CA certificates are in here (e.g. download.opensuse.org uses Let's Encrypt), so I don't understand how HTTPS connections are even supposed to work correctly at all.

[salfers]

After deleting the SSVM and letting Cloudstack recreate it the normal certificates are now in the keystore:

root@s-325-VM:/usr/local/cloud/systemvm# keytool -list -keystore certs/realhostip.keystore
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 136 entries

cross, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): 18:F8:A7:A1:51:B4:EC:28:08:98:09:3D:F5:BD:53:7C:A0:99:CC:27:74:05:D0:28:1D:E0:DA:DF:D1:44:20:DA
debian:ac_raiz_fnmt-rcm.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): EB:C5:57:0C:29:01:8C:4D:67:B1:AA:12:7B:AF:12:F7:03:B4:61:1E:BC:17:B7:DA:B5:57:38:94:17:9B:93:FA
debian:accvraiz1.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 9A:6E:C0:12:E1:A7:DA:9D:BE:34:19:4D:47:8A:D7:C0:DB:18:22:FB:07:1D:F1:29:81:49:6E:D1:04:38:41:13
debian:actalis_authentication_root_ca.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 55:92:60:84:EC:96:3A:64:B9:6E:2A:BE:01:CE:0B:A8:6A:64:FB:FE:BC:C7:AA:B5:AF:C1:55:B3:7F:D7:60:66
debian:affirmtrust_commercial.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 03:76:AB:1D:54:C5:F9:80:3C:E4:B2:E2:01:A0:EE:7E:EF:7B:57:B6:36:E8:A9:3C:9B:8D:48:60:C9:6F:5F:A7
debian:affirmtrust_networking.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 0A:81:EC:5A:92:97:77:F1:45:90:4A:F3:8D:5D:50:9F:66:B5:E2:C5:8F:CD:B5:31:05:8B:0E:17:F3:F0:B4:1B
debian:affirmtrust_premium.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 70:A7:3F:7F:37:6B:60:07:42:48:90:45:34:B1:14:82:D5:BF:0E:69:8E:CC:49:8D:F5:25:77:EB:F2:E9:3B:9A
debian:affirmtrust_premium_ecc.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): BD:71:FD:F6:DA:97:E4:CF:62:D1:64:7A:DD:25:81:B0:7D:79:AD:F8:39:7E:B4:EC:BA:9C:5E:84:88:82:14:23
debian:amazon_root_ca_1.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 8E:CD:E6:88:4F:3D:87:B1:12:5B:A3:1A:C3:FC:B1:3D:70:16:DE:7F:57:CC:90:4F:E1:CB:97:C6:AE:98:19:6E
debian:amazon_root_ca_2.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 1B:A5:B2:AA:8C:65:40:1A:82:96:01:18:F8:0B:EC:4F:62:30:4D:83:CE:C4:71:3A:19:C3:9C:01:1E:A4:6D:B4
debian:amazon_root_ca_3.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 18:CE:6C:FE:7B:F1:4E:60:B2:E3:47:B8:DF:E8:68:CB:31:D0:2E:BB:3A:DA:27:15:69:F5:03:43:B4:6D:B3:A4
debian:amazon_root_ca_4.pem, May 22, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): E3:5D:28:41:9E:D0:20:25:CF:A6:90:38:CD:62:39:62:45:8D:A5:C6:95:FB:DE:A3:C2:2B:0B:FB:25:89:70:92
[...]

So it seems there is a bug where these are missing under some conditions.

[shwstppr]

@salfers thanks for the update. Yes, it could be a bug but I'm not sure of the reproduction steps.
In all my test env keystores have the entries.

root@s-1-VM:/usr/local/cloud/systemvm# keytool -list -keystore certs/realhostip.keystore 
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 136 entries
...

[Dimonyga]

After June 6, downloading images from sources with certificates issued by letsencrypt stopped working.
https://letsencrypt.org/2024/04/12/changes-to-issuance-chains

[midhunpjos]

+1 Facing same issue with 4.19.1 KVM+EDGE ZONE
2024-08-08 12:58:42,020 INFO [resource.wrapper.LibvirtCheckUrlCommand] (agentRequest-Handler-2:null) (logid:f0f98208) Checking URL: https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img, with connect timeout: 5000, connect request timeout: 5000, socket timeout: 5000
2024-08-08 12:58:43,018 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:f0f98208) Caught:
com.cloud.utils.exception.CloudRuntimeException: Cannot obtain qcow2 virtual size due to: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.apache.cloudstack.direct.download.HttpsDirectTemplateDownloader.getRemoteFileSize(HttpsDirectTemplateDownloader.java:199)
at org.apache.cloudstack.direct.download.DirectDownloadHelper.getFileSize(DirectDownloadHelper.java:99)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtCheckUrlCommand.execute(LibvirtCheckUrlCommand.java:47)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtCheckUrlCommand.execute(LibvirtCheckUrlCommand.java:30)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1929)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:360)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:298)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1613)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1541)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at java.base/java.net.URL.openStream(URL.java:1165)
at org.apache.cloudstack.direct.download.HttpsDirectTemplateDownloader.getRemoteFileSize(HttpsDirectTemplateDownloader.java:197)
... 13 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341)
... 31 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)

[salfers]

Issue happened again, this time on 4.19.1.1.
Situation looks the same as above:

root@s-325-VM:/usr/local/cloud/systemvm# keytool -list -keystore certs/realhostip.keystore
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 4 entries

cross, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): 18:F8:A7:A1:51:B4:EC:28:08:98:09:3D:F5:BD:53:7C:A0:99:CC:27:74:05:D0:28:1D:E0:DA:DF:D1:44:20:DA
intermed, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): 09:ED:6E:99:1F:C3:27:3D:8F:EA:31:7D:33:9C:02:04:18:61:97:35:49:CF:A6:E1:55:8F:41:1F:11:21:1A:A3
realhostip, Feb 3, 2012, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): ED:76:FC:C1:8A:84:B6:C6:AC:88:FE:9C:F0:22:B2:9D:83:54:7E:5B:5C:92:0B:10:B7:D5:70:27:92:1C:7B:EF
root, Feb 3, 2012, trustedCertEntry, 
Certificate fingerprint (SHA-256): C3:84:6B:F2:4B:9E:93:CA:64:27:4C:0E:C6:7C:1E:CC:5E:02:4F:FC:AC:D2:D7:40:19:35:0E:81:FE:54:6A:E4

No clue on reproduction steps unfortunately. The SSVM has been running for 120 days, so it's not fresh.