How to properly configure gRPC mTLS with pkcs11-provider?

[mykola-kobets-epam]

Hello All,

I'm working on setting up gRPC mTLS connections with pkcs11-provider and SoftHSM, and I've run into some issues. I'd appreciate any guidance you could provide.

My setup:

• gRPC (C++) 1.60.1
• OpenSSL 3.2.6
• pkcs11-provider 1.0.0
• SoftHSM 2.6.1

openssl.cnf has a minimal pkcs11-provider configuration:

[pkcs11_sect]
module = /usr/lib/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/lib/softhsm/libsofthsm2.so
activate = 1

I've tried two approaches:

First approach: I'm using CA and client certificate chains exported from SoftHSM in PEM format, with the private key provided via a pkcs11-provider URI. I expected OpenSSL to use the default provider for operations that don't require the private key, and only switch to pkcs11-provider when the private key is needed. The initial mTLS connection works, but reconnections fail because OpenSSL caches the pkcs11-provider after the first connection and then uses it for other mTLS operations, not just private key operations.

Second approach: I tried providing all certificates as pkcs11-provider URIs, but that didn't work. It seems PEM_read_bio_X509_AUX (which gRPC uses to read certificates) doesn't use the provider API, also I'm not sure whether pkcs11-provider implements decoder for reading certificates.

Any suggestions would be very helpful. Thank you for your time!

[simo5]

I expected OpenSSL to use the default provider for operations that don't require the private key, and only switch to pkcs11-provider when the private key is needed. The initial mTLS connection works, but reconnections fail because OpenSSL caches the pkcs11-provider after the first connection and then uses it for other mTLS operations, not just private key operations.

This is what should happen, please provide some logs to show what else is happening.
Keep in mind that SoftHSM is not a good to ken to do testing with, besides having protocol implementation issues, it links directly to OpenSSL in a bad way (uses default library context) sometimes causing deadlocks and loops.
I suggest using a different token like kryoptic.

[mykola-kobets-epam]

Please find pkcs11-provider + tsi grpc traces below.
pin-for-pkcs11.problem.txt

There are two client gRPC connections to different services:

Connection 1 handshake:
I0225 16:43:40.083215164 1327 ssl_transport_security.cc:221] HANDSHAKE START - before SSL initialization - PINIT
I0225 16:43:40.093522723 1338 ssl_transport_security.cc:221] LOOP - SSLv3/TLS read server session - TRST

Connection 2 handshake:
I0225 16:43:40.923170358 1340 ssl_transport_security.cc:221] HANDSHAKE START - before SSL initialization - PINIT
I0225 16:43:41.005263885 1340 ssl_transport_security.cc:221] LOOP - SSLv3/TLS read server session - TRST

After Connection 2 is restarted, the handshake fails in the TRCR state:
I0225 16:44:00.200894270 1340 ssl_transport_security.cc:221] HANDSHAKE START - before SSL initialization - PINIT
I0225 16:44:01.209581118 1338 ssl_transport_security.cc:221] LOOP - SSLv3/TLS read server certific - TRCR

At the end pkcs11-provider prints "Enter PIN for PKCS#11 Token" message while trying to open a session.

I also attached a stack trace below up to p11prov_session_prompt_for_pin and list of available slots.
pin-for-pkcs11.bt.txt
softhsm-slots.txt

Concerning

OpenSSL in a bad way (uses default library context) sometimes causing deadlocks and loops.

Thank you for the hint. I haven't seen any problems with softhsm for now, but I'll try to integrate kryoptic into my test env.