Quick question about keystore (jks) requirement

[dcontiveros-nf]

Hello fellow cloudstack users/admins.

I had a quick question concerning JKS requirements. From what I understand, this is the join procedure from an agent's viewpoint:

1. The agent communicates to the management server on port 8250,
2. A certificate sent over the handshake and entries added to cloud.jks.
3. Libvirt transfers will only work for this:

Starting 4.11.1, a KVM host is considered secured when it has its keystore and certificates setup for both the agent and libvirtd process. A secured host will only allow and initiate TLS enabled live Instance migration. This requires libvirtd to listen on default port 16514, and the port to be allowed in the firewall rules

Is there a way to disable this functionality? We are in PoC stage and starting to fully automate a lot of these prereqs. We have some custom tooling around .jks generation, but am just wondering if this part is mandatory for functionality.

Thanks!

[rajujith]

@dcontiveros-nf I believe setting the value for global configuration ca.plugin.root.auth.strictness should help. cc @weizhouapache

[dcontiveros-nf]

@rajujith That variable did not work for connecting to CS Agent -> CS Management mode. I think we need to take a step back now that I have some more information.

We are trying to add nodes out of band, meaning we want to set them up in an automatic fashion with no access to GUI. This shouldn't be a problem as the docs specifically state that we would need to run some commands here(Personal opinion, I believe this needs to be rewritten so as to not conflict with the Out-Of-Band management that is brought up earlier in that doc.
):

https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/hosts.html#securing-process

We attempted to replicate this procedure utilizing only key-tool, but could not get this to connect. I also searched the docs for any PKI related information that would possibly point to how to properly do this, but I did not find anything. What I need now is some direction to do the following:

1. Properly setup the JKS stores for both manager/agent that will result in a successful connection without using scripts and either openssl or key-tool
2. Possible way to increase logging for NIO since it handles SSL handshake and is where my agent is getting killed

Ideally I think it would be amazing if ACS could do:

1. Update docs with a PKI section
2. Update the docs with alternative methods to handle PKI generation
3. Diagrams (haha I know, I love a picture)

I can provide any followups necessary. If I need to enter a Bug/Feature request I can also do that.

Edit: I am aware I could setup a MVP, dump the keystores, and then replicate what gets actioned. However, this doesn't guarantee that the procedure will break later on on upgrades. I rather avoid going this route.

[dcontiveros-nf]

A few updates here on where I am at in this process:

Goal
To be able to automate deployment of KVM nodes

Blocker
I cannot seem to get the proper procedure in place to automate generation of a compatible keystore so CloudStack manager node can allow auth on it

Investigation
To get past this blocker, I attempted to find out if I could replicate the proper format of the JKS keystore that Cloudstack expects. Here is where I see the first issue.

Issue 1
After looking at the code, it appears that Cloudstack expects cloud.jks to be used, so I can only assume that it is not possible to separate keystore and truststore JKS files the way most Java apps are designed to do. That's ok. We can always concatenate and format the relevant certs and insert into the JKS file in the proper order. Still, this requires knowing how to setup the JKS file. I do believe it is possible to get something the management node expects. I looked at the mailing lists and saw a post make a reference to an issue that was opened back in 2020 with a similar goal. Here is the relevant issue and comments:

#4199 (comment)
#4199 (comment)

It appears this particular individual is automating creation of a PKCS#12 file with their certs, converting that to a JKS store, and then configuring Netty/Nio to use it. I attempted to create a Netty compatible JKS keystore with the following order (same as the comment):

PrivateKey
  Leaf Cert
  Intermediate
  Root CA

We have an existing pfx bundle and I did check the keystore had the above order but I still received SSL handshake errors via Nio on the Agent side:

# KVM node
2025-05-01 17:02:02,589 INFO  [cloud.agent.Agent] (main:[]) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again... com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: HOST REDACTED: 8250

# Management node
2025-05-01 17:03:47,848 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:[]) (logid:) SSL error caught during wrap data: Empty client certificate chain, for local address=/REDACTED:8250, remote address=/REDACTED:44838.

That last part of the management node is of relevance because of this issue:

#5805

This implies that adding a KVM node is necessary via GUI, but we are attempting to automate this completely. Specifying ca.plugin.root.auth.strictness=false is probably not the way to go here since our team will be deploying numerous nodes. We have deployed the agent successfully, but are missing the proper format of the JKS file. Searching for Jetty or Nio didn't reveal much for the latest docs.

Issue 2
It appears that Cloudstack does not differentiate between truststores for https connectivity to the GUI and traffic flowing between management nodes and agent nodes. I searched the entire project for JKS and my IDE returned only instances to cloud.jks or whatever is defined as the JKS file in the variable https.keystore. I'm not certain if this is used for both Netty and Nio, but it appears as if it is. Please let me know if I am mistaken as I only took a cursory glance. This isn't really a MAJOR issue, but there may be some desire to seperate out these two types of traffic.

Issue 3
This is specific to KVM. It appears that helper scripts are used to register the KVM hypervisor node to the management agent. Docs here point to this:

When a new host is being setup, such as adding a KVM host or starting a systemvm host, the CA framework kicks in and uses ssh to execute keystore-setup to generate a new keystore file cloud.jks.new, save a random passphrase of the keystore in the agent’s properties file and a CSR cloud.csr file. The CSR is then used to issue certificate for that agent/host and ssh is used to execute keystore-cert-import to import the issued certificate along with the CA certificate(s), the keystore is that renamed as cloud.jks replacing an previous keystore in-use. During this process, keys and certificates files are also stored in cloud.key, cloud.crt, cloud.ca.crt in the agent’s configuration directory.

It appears the code to do this is performed with the utility scripts in this path:

scripts/util/keystore-cert-import
scripts/util/keystore-setup

Upon inspecting this file it appears that on registration, the manager will SSH into the host, run this script, setup a cloud.jks, generate a CSR for signing, and then inject that cert into the keystore. I see that it uses some hard coded attributes as well. I searched for any info relevant to interacting with the management node for signing a CSR, but there is almost no information in the docs. This makes me suspect I will run into the same issue when I call a REST API endpoint that either relies on a SystemVM or generates one. This is a bit alarming since we believe we will need to deploy some zones that may require RouterVMs. We are not certain as I cannot get this far into the automation or even registering one node. This part is possibly related to the Issue 1 but chose to show its own issue since it is clearly stated in the docs that this procedure happens with SystemVMs (although not with that particular helper script).

Issue 4
It appears the alias used when successfully storing information is Cloud. The helper script scripts/util/keystore-setup is a bash script, but when it is called the ALIAS variable isn't sourced from an argument but rather hardcoded in the script, making this inflexible. I had setup a single node cluster (one testbench machine) to get used to cloudstack UI prior to this endeavour. Luckily I still had access to this file. I see the following when I dump all the info in keytool:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

Alias name: cloud
Entry type: PrivateKeyEntry

Alias name: cloudca.1
Entry type: trustedCertEntry

This matches code in:

scripts/util/keystore-cert-import

This makes me believe that prior to deployment we need to stage valid certs with additional aliases based on our Root CA for all agents/hypervisor nodes. Now again, this JKS file was from a single node cluster. I understand there may be some differences in how the JKS will ultimately look in a distributed fashion.

Conclusion
We are running into quite a bit of issues with authentication and would like to resolve these ASAP. Since we have our own desired PKI, this is making things a bit difficult for us since documentation is a bit lax on this topic.

Any feedback is appreciated. I did post on Slack and saw these messages get sent out to the mailing list, so any help would be appreciated.

[DaanHoogland]

@dcontiveros-nf I don not understand the full extend of your issue but

ad issue 1:

This implies that adding a KVM node is necessary via GUI, but we are attempting to automate this completely.

The GUI is completely operating by API-calls so anything you can do in the GUI, you can snoop the APIs for and incorporate them in your scripts.

ad issue 2:

Yes, I think you are right. Your mild requirement would ask for a major code change I think.

In general , have a look at https://www.shapeblue.com/cloudstack-ca-framework/ and at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+Agent+Communications (particularly the remark on writing your own #SecureAgentCommunications-DIYCAProviderPlugin)

I am not sure if this is what you are looking for, but it might give some extra insights.

[weizhouapache]

@dcontiveros-nf

currently when provision a host key, cloudstack will generate a cert/key/jks for cloudstack-agent.

/etc/cloudstack/agent/cloud.ca.crt
/etc/cloudstack/agent/cloud.crt
/etc/cloudstack/agent/cloud.key

these are also used by libvirt . you may have noticed in the script scripts/util/keystore-cert-import

• if you want to use self-signed cert for libvirt

it is easy, you just need to remove the symblink and use your own key.
do not forget to remove the ln -sf in the script scripts/util/keystore-cert-import

• if you want to use self-signed cert for cloudstack-agent

I think it is possible (but I never tested it)
please bear in mind that the cert/key/jks are generated with a RSA key and a CA certificate which are generated by ACS, you cannot use your own.

you can find the RSA public/private key and ca certificate by DB query

SELECT name,value FROM configuration WHERE name in ('ca.plugin.root.ca.certificate', 'ca.plugin.root.public.key', 'ca.plugin.root.private.key');

these values are encrypted by db secret key, you can decrypt the value, please refer to
https://cwiki.apache.org/confluence/display/CLOUDSTACK/New+database+encryption+cipher+-+AeadBase64Encryptor#NewdatabaseencryptioncipherAeadBase64Encryptor-6.EncryptionCLIincloudstack-utils.jar

• Alternatively, you can add the host at first, then provision a host key

ca.plugin.root.auth.strictness should be false