Server Loading Certs from file works, but not via thumbprint from Windows Cert Store

[barafael]

We are trying to connect a server and a client via TLS with self-signed certs. For this purpose we have generated the certificates as detailed in the page https://docs.nats.io/running-a-nats-service/configuration/securing_nats/tls .

What Works

When we load the certs by cert_file and key_file, it works. The server and client talk just fine.

What doesn't Work

We create a certificate bundle using

openssl pkcs12 -export -out certificate.pfx -inkey server.key -in server.crt

When we load that certificate via thumbprint from the windows certificate store, it does not work. We have verified that loading succeeds because if the thumbprint (or subject) are wrong, loading fails. We have traced that the nats server goes beyond the point of loading the certs.

When the connection is being established, we get BadSignature errors on the client side

DEBUG ThreadId(02) async_nats::connector: connection attempt failed server=ServerAddr(Url { scheme: "tls", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("172.31.63.27")), port: Some(4222), path: "", query: None, fragment: None }) error=IO error: invalid peer certificate: BadSignature

and on the server side:

[9128] 2025/08/21 17:02:20.919845 [←[31mERR←[0m] 172.31.63.219:52192 - cid:15 - TLS handshake error: remote error: tls: error decrypting message

In wireshark, we see that a tls connection attempt is made but ends early.

We see that the difference between the two executions is the different crypto.Signer interfaces winKey and an RSA key. The certificate data are equal, at least.

Question

Are we holding it wrong, or could it be there is an issue in nats-server?

[bkaestner]

Are we holding it wrong, or could it be there is an issue in nats-server?

Most-likely an issue in the cert-store specific code path for certificate signatures, see #7306.