[EJBCA-CE/Wildfly 26] Unable to create HSM-backed crypto token when running as systemd service

[sbarnes-tecnica]

Hello!

I've been working on getting EJBCA-CE running with Wildfly 26 and using a YubiHSM2 via PKCS11 as a secure keystore.
My issue is that when I'm done, EJBCA correctly detects that the YubiHSM2 PKCS11 library is present, but as soon as I try to create the token, I'm presented with an error

Error: Error when creating Crypto Token with ID 1683674145

Even upping the logging to TRACE using

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as.config:write-attribute(name=level, value=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss:add(level=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.wildfly:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.wildfly:add(level=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.xnio:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.xnio:add(level=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate:add(level=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf:add(level=TRACE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.config.ConfigurationHolder:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.config.ConfigurationHolder:add(level=TRACE)'

didn't reveal anything more, however when trying the same thing in Docker, I was able to find the following line in the resulting stack trace:

Caused by: java.io.IOException: libyubihsm.so.2: cannot open shared object file: No such file or directory/usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so

Now this is strange, as the library is definitely installed:

ejbca-admin@ejbca:~$ ls -lah /usr/lib/x86_64-linux-gnu/pkcs11
total 392K
drwxr-xr-x 1 wildfly wildfly  138 Mar 26 15:11 .
drwxr-xr-x 1 root    root     21K Mar 26 15:11 ..
lrwxrwxrwx 1 wildfly wildfly   26 Mar 10  2022 onepin-opensc-pkcs11.so -> ../onepin-opensc-pkcs11.so
lrwxrwxrwx 1 wildfly wildfly   19 Mar 10  2022 opensc-pkcs11.so -> ../opensc-pkcs11.so
lrwxrwxrwx 1 wildfly wildfly   16 Mar 10  2022 pkcs11-spy.so -> ../pkcs11-spy.so
-rw-r--r-- 1 wildfly wildfly 380K Oct 30 13:55 yubihsm_pkcs11.so

Stopping Wildfly and running the following directly as the service user resolves the issue
sudo -u wildfly /opt/wildfly/bin/launch.sh standalone standalone.xml 0.0.0.0

Wildfly is present in /opt/wildfly as a symlink to /opt/wildfly-26.1.3.Final
EJBCA is installed to /opt/ejbca

Wildfly is running with the following unit file:

[Unit]
Description=The WildFly Application Server
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
EnvironmentFile=-/etc/wildfly/wildfly.conf
User=wildfly
LimitNOFILE=102642
PIDFile=/var/run/wildfly/wildfly.pid
ExecStart=/opt/wildfly/bin/launch.sh $WILDFLY_MODE $WILDFLY_CONFIG $WILDFLY_BIND
StandardOutput=null

[Install]
WantedBy=multi-user.target

/etc/wildfly/wildfly.conf

# The configuration you want to run
WILDFLY_CONFIG=standalone.xml

# The mode you want to run
WILDFLY_MODE=standalone

# The address to bind to
WILDFLY_BIND=0.0.0.0

#Add native library path for PKCS11
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
LD_LIBRARY_PATH="/lib:/usr/lib"

Given that this issue only occurs when running under systemd, some kind of path/environment issue is all I can think of, but even putting all of user wildfly's environment variables into /etc/wildfly/wildfly.conf doesn't resolve the issue.

[sbarnes-tecnica]

The issue was one of environment variables. After further debugging, the wildfly systemd unit was not picking up
YUBIHSM_PKCS11_CONF=/etc/yubihsm2/yubihsm_pkcs11.conf
from the environment.

Manually adding this to /etc/wildfly/wildfly.conf resolved the issue.

I suspect this would work with any HSM that runs into this issue, but I only have a YubiHSM2 to test with.