Debian 13, OpenSSL 3.5.4 - crash when using pkcs11 engine

[drauch]

We use openssl cms in combination with your module to perform the signing operation from our PKCS#11 module.

On all other OS we test on (Debian 11, 12, Ubuntu 24, RedHat 8, 9) this works fine. However, on Debian 13 the log of our PCKS#11 library is longer: it contains the following additional PKCS#11 calls:

2025-11-29 09:23:10,969 [744:1] DBG <> EntryPoint - C_CloseAllSessions called: slotId=1
2025-11-29 09:23:10,971 [744:1] DBG <> EntryPoint - C_CloseAllSessions returns with 0
2025-11-29 09:23:10,971 [744:1] DBG <> EntryPoint - C_Finalize called: isnull(pReserved)=true
2025-11-29 09:23:10,972 [744:1] DBG <> EntryPoint - C_Finalize returns with 0

Afterwards we get a segmentation fault at this stack trace:

Thread 1 "openssl" received signal SIGSEGV, Segmentation fault.
0x00007bce2150db30 in ?? ()
(gdb) bt
#0  0x00007bce2150db30 in ?? ()
#1  0x00007bce22f1e93a in CRYPTO_free_ex_data (class_index=class_index@entry=3, obj=obj@entry=0x5abce851d770, ad=ad@entry=0x5abce851d838) at ../crypto/ex_data.c:406
#2  0x00007bce230a9f04 in x509_cb (operation=<optimized out>, pval=<optimized out>, it=<optimized out>, exarg=<optimized out>) at ../crypto/x509/x_x509.c:85
#3  0x00007bce22de4212 in ossl_asn1_item_embed_free (pval=pval@entry=0x5abce8565ac8, it=0x7bce23289040 <local_it>, embed=embed@entry=0) at ../crypto/asn1/tasn_fre.c:117
#4  0x00007bce22de43c3 in ossl_asn1_template_free (pval=0x5abce8565ac8, tt=tt@entry=0x7bce232b5340 <CMS_CertificateChoices_ch_tt>) at ../crypto/asn1/tasn_fre.c:146
#5  0x00007bce22de4289 in ossl_asn1_item_embed_free (pval=pval@entry=0x7ffc0d668770, it=<optimized out>, embed=embed@entry=0) at ../crypto/asn1/tasn_fre.c:70
#6  0x00007bce22de4365 in ossl_asn1_template_free (pval=0x5abce8501958, tt=tt@entry=0x7bce232b5018 <CMS_SignedData_seq_tt+120>) at ../crypto/asn1/tasn_fre.c:141
#7  0x00007bce22de41f3 in ossl_asn1_item_embed_free (pval=pval@entry=0x5abce8501918, it=0x7bce2325f640 <local_it>, embed=embed@entry=0) at ../crypto/asn1/tasn_fre.c:114
#8  0x00007bce22de43c3 in ossl_asn1_template_free (pval=0x5abce8501918, tt=tt@entry=0x7bce232b4118 <CMS_ContentInfo_adbtbl+56>) at ../crypto/asn1/tasn_fre.c:146
#9  0x00007bce22de41f3 in ossl_asn1_item_embed_free (pval=pval@entry=0x7ffc0d6688c8, it=0x7bce2325f0c0 <local_it>, embed=embed@entry=0) at ../crypto/asn1/tasn_fre.c:114
#10 0x00007bce22de42c9 in ASN1_item_free (val=<optimized out>, val@entry=0x5abce8501910, it=<optimized out>) at ../crypto/asn1/tasn_fre.c:20
#11 0x00007bce22e45da9 in CMS_ContentInfo_free (a=a@entry=0x5abce8501910) at ../crypto/cms/cms_lib.c:27
#12 0x00005abcd6ae2ab4 in cms_main (argc=<optimized out>, argv=<optimized out>) at ../apps/cms.c:1320
#13 0x00005abcd6afa991 in do_cmd (prog=prog@entry=0x5abce85038e0, argc=argc@entry=17, argv=argv@entry=0x7ffc0d668d00) at ../apps/openssl.c:428
#14 0x00005abcd6acfb46 in main (argc=<optimized out>, argv=<optimized out>) at ../apps/openssl.c:309

It sounds like (only a wild guess by myself) there is some ex_data cleanup that happens after the PKCS#11 module has already been unloaded?

Hope I can help to fix this issue soon, we're not able to deliver our PKCS#11 module for Debian 13 at the moment without statically linking an outdated OpenSSL version.

Best regards,
D.R.

[mtrojnar]

Is this issue reproducible with the latest libp11 release and the latest Git master branch?

If the issue is not reproducible on the latest release, it should be reported to the package maintainer and not to the upstream.

[drauch]

How can I replace the Debian 13 version with the lastest libp11 release version in an easy way? I'm no Linux expert.

Best regards,
D.R.

[ulrichb]

@mtrojnar I did a bit of debugging and have a few inputs here.

(All tests ran on an current Debian 13 Docker image with OpenSSL 3.5; Debian package version "3.5.4-1~deb13u1".)

• We get the same seg fault and same stack trace as @drauch posted above with the current master version as well as with the tags libp11-0.4.14 and libp11-0.4.13.
  • The seg fault always happens after (successful) calls of C_CloseAllSessions() and C_Finalize().
  • But: tag libp11-0.4.12 works!
• The issue only happens after performing an (RSA) signing operation in the openssl cms command . We also tested openssl dgst and openssl pkeyutl and there it works fine (in any version).
• When we comment out the "ctx_finish()" in engine_finish() (in v0.4.13) the segfault also disappears.
  • But this may be a red herring, because we saw that the engine_finish() function does not get called at all in 0.4.12, see the log screenshot below, where I added printf loggings for get_ctx(), engine_init(), engine_destroy(), and engine_finish().

Printf logging of openssl cms in 0.4.12 vs. 0.4.13 (both with OpenSSL 3.5.4):

[dengert]

As pointed out by @ulrichb 30 mins ago;

Debian 13 libp11 is based on libp11 0.4.13 https://packages.debian.org/source/trixie/libp11

libp11 is at 0.4.16 https://github.com/OpenSC/libp11

https://github.com/OpenSC/libp11/releases/tag/libp11-0.4.14 introduced using OpenSSl 3 providers reated then older OpenSSl engine which did use "ex_data". The provider does need to do this.

Debian needs to be updated to use newer libp11 source. You can turn in a bug report on https://packages.debian.org/source/trixie/libp11.

[drauch]

OpenSSL 3 providers are "not yet there yet", for now all distros use the engine model still. I don't think that the switch will happen before the next major distro updates are coming out.

For now we should still fix the OpenSSL engine approach, don't you think? It would be premature to drop support for the engine approach completely?

Best regards,
D.R.

[ulrichb]

@dengert Debian doesn't do feature updates in their stable versions, only sec/bugfix patches. (Note that they already updated libp11 in their unstable "sid" to v0.4.16.)

The problem we're trying to solve here is to get the libp11 engine working under the stable Debian 13 (which will be the latest stable for 1.5 more years) for CMS signing.

Also in future Debian versions we may require to stick to the engine because of downstream tools which use OpenSSL to perform CMS signing which still only support engines (e.g. we saw that in some proprietary firmware signing tools). And engine support will stay as long OpenSSL is 3.x is supported, doesn't it? (I know, it's annoying that we're in this transition phase 😞.)

... and in this scenario also 0.4.16 isn't enough, as shown above in the OpenSSL cms tests with the engine.

P.S. (maybe important context): we don't use the engine/provider by ourselves, but our users and therefore don't have the package versions/OS/calling tools/... under our control.

[mtrojnar]

Feel free to make the trivial patch that removes the cleanup function calls and submit the patch to the Debian package maintainer. While package maintainers are generally not allowed to deploy new package versions, they may still apply trivial patches and release patched code of the current version.

There is nothing we can do about it here, upstream.

[mtrojnar]

How can I replace the Debian 13 version with the lastest libp11 release version in an easy way? I'm no Linux expert.

sudo apt install libssl-dev

Unpack the libp11 tarball and

./configure && make && sudo make install

[drauch]

Unpack the libp11 tarball and

Thank you, I now see the same results as @ulrichb does.

Feel free to make the trivial patch that removes the cleanup function calls and submit the patch to the Debian package maintainer. While package maintainers are generally not allowed to deploy new package versions, they may still apply trivial patches and release patched code of the current version.

There is nothing we can do about it here, upstream.

@mtrojnar: Why are you so sure that there is nothing that can be done in this repo? v0.4.12 still works and v0.4.13 doesn't. Doesn't it suggest itself that there has been a breaking change / bug introduction in libp11 from 0.4.12 to 0.4.13 which now renders multiple clients (e.g., openssl cms, osslsigncode) unuseable?

Best regards,
D.R.

[dengert]

I would suggest that you submit a bug report to Debian. When you say it is versions are v0.4.12 and v0.4.13 are there any changes applied by debian?
For example:
https://packages.debian.org/trixie/libp11-dev
has changes so version is actually 0.4.13-1 [libp11_0.4.13-1.debian.tar.xz]
Which leads to https://bugs-devel.debian.org/cgi-bin/bugreport.cgi?bug=1079985
that backported https://salsa.debian.org/opensc-team/libp11/-/commit/4876e07c4968c93c38ce1375a0ef51bc2e3c9b27

The above is not the RSA issue, but similar.

[mtrojnar]

Thank you, I now see the same results as @ulrichb does.

Which results exactly? Could you be a bit more specific?

@mtrojnar: Why are you so sure that there is nothing that can be done in this repo?

It's because Debian policy won't allow updating their libp11 package in Debian 13 to any other version of this repo.

v0.4.12 still works and v0.4.13 doesn't. Doesn't it suggest itself that there has been a breaking change / bug introduction in libp11 from 0.4.12 to 0.4.13 which now renders multiple clients (e.g., openssl cms, osslsigncode) unuseable?

Correct. Those old versions had a bug. How is it relevant?

[dengert]

Rather then removing the ctx_finish(); use this commit feom last year:
3873b45

[drauch]

Which results exactly? Could you be a bit more specific?

Segmentation faults in newer versions, working in old versions.

Correct. Those old versions had a bug. How is it relevant?

No, you misread ulrichb/my posts: the old version doesn't have the bug, the new version has the bug. 0.4.12 works BUT up-to-date main does NOT work.

For example:
https://packages.debian.org/trixie/libp11-dev
has changes so version is actually 0.4.13-1 [libp11_0.4.13-1.debian.tar.xz]

We checked out the tag from this repository and built from scratch and used this version for testing instead of the Debian version. Still the problem with the newest main.

Best regards,
D.R.

[mtrojnar]

Good to know. The report does not clearly state which versions are affected. Please consider reading How to Report Bugs Effectively to prevent future misunderstandings.

Please update the issue description to include a step-by-step guide to reproduce the bug. Preferably, submit a PR with a test, so that this bug is tested automatically in future releases.

Does the issue occur with that specific version of OpenSSL, or only with the Debian's OpenSSL package?

[ulrichb]

The report does not clearly state which versions are affected.

I tried to deliver this info in the reply #629 (comment) (including a few other observations).

Additional new observations on Ubuntu (all tests with openssl cms and the engine):

• Ubuntu 24.04 w/ OpenSSL 3.0.13, libp11 0.4.12-1.1build2: works fine
• Ubuntu 25.04 w/ OpenSSL 3.4.1, libp11 0.4.13-1: seg fault
• Ubuntu 26.04 (current preview) w/ OpenSSL 3.5.3, libp11 0.4.16-2: seg fault

So similar to Debian 13 when having libp11 > 0.4.12, we seem to have an issue with newer OpenSSL versions (issue starts somewhere between ]3.0.13, 3.4.1]).

When you say it is versions are v0.4.12 and v0.4.13 are there any changes applied by debian?

See my tests in #629 (comment): also when building libp11 on Debian 13 from source (origin repo) we get the seg faults => not a Debian patch issue.