Add support for HMAC

[ilie-halip-nxp]

Description

In the context of TLS 1.3, HMAC is used to calculate the verify_data with the Finished key. Right now, OpenSSL falls back to it's own internal HMAC.

Add an HMAC implementation that calls into the PKCS11 library to do the HMAC calculation. TLS1.3 does not truncate the HMAC length, so use CKM_XXX_HMAC mechanisms without an explicit length.

Add tests that compare the output from PKCS11 to the output of OpenSSL. Since PKCS11 libs may not have implementations for all the mechanisms in the specification, define a subset of "mandatory" HMACs that should be supported by all the test suites.

In the process, I had to add 2 new attributes for the generic secret key:

• CKA_SIGN: needed to be able to call C_SignInit with the key
• CKA_PRIVATE: softhsm seems to require this attribute for everything to work fine - I guess this could be removed, but the tests would have to be disabled for softhsm

Checklist

• Code modified for feature
• Test suite updated with functionality tests

Reviewer's checklist:

• Any issues marked for closing are addressed
• There is a test suite reasonably covering new functionality or modifications
• This feature/change has adequate documentation added
• Code conform to coding style that today cannot yet be enforced via the check style test
• Commits have short titles and sensible commit messages
• Coverity Scan has run if needed (code PR) and no new defects were found

[simo5]

This looks like good work, and I have mostly nits, however I got to ask why do you care to execute the HMAC function on the token, given you are not using really a token generated key, but rather importing a key that openssl has access to already.

[ilie-halip-nxp]

Hey Simo, thanks for the quick review!

why do you care to execute the HMAC function on the token, given you are not using really a token generated key, but rather importing a key that openssl has access to already.

This is just to allow delegating the operation to a token that is a hardware crypto accelerator. But, I have some WIP changes to do TLS related operations entirely on the token, so this would be a prerequisite for those to work. Do you think such changes could be accepted, in future PRs?

[simo5]

Hey Simo, thanks for the quick review!

why do you care to execute the HMAC function on the token, given you are not using really a token generated key, but rather importing a key that openssl has access to already.

This is just to allow delegating the operation to a token that is a hardware crypto accelerator. But, I have some WIP changes to do TLS related operations entirely on the token, so this would be a prerequisite for those to work. Do you think such changes could be accepted, in future PRs?

We are interested in that kind of functionality, yes, although it may require new functionality in OpenSSL as well. But we can collaborate in getting what's needed there, as we are also expanding on the EVP_SKEY stuff (which MAC code should really be able use once we add EVP_SKEY to MAC operations in OpenSSL)

[simo5]

A few more nits, but overall this looks good

[ilie-halip-nxp]

Hey, I believe I went through all the comments.

I see there are a couple of CI failures, maybe these are sporadic failures but I'm taking a look. Any other things which would I need to address before merging?

[Jakuje]

I see there are a couple of CI failures, maybe these are sporadic failures but I'm taking a look. Any other things which would I need to address before merging?

I think the failures are related to to openssl or something. Please, rebase your changes on top of current main.

[simo5]

Very close, just some minor nits.

[simo5]

Early loading seem odd to need, can you instead force all operations on the token? Otherwise there is no guarantee thes operations are done on the token during tests.
See other tests for the properties to set to force all ops on a token.

[ilie-halip-nxp]

Forcing all ops on the token leads to a hang because of a seemingly non-recursive mutex used by SoftHSM to lock the token. Here's a backtrace:

  * frame #0: 0x00007ffff7298f70 libc.so.6`__GI___lll_lock_wait [inlined] futex_wait(private=0, expected=2, futex_word=0x00005555556e89e0) at futex-internal.h:146:13
    frame #1: 0x00007ffff7298f5a libc.so.6`__GI___lll_lock_wait(futex=0x00005555556e89e0, private=0) at lowlevellock.c:49:7
    frame #2: 0x00007ffff72a0101 libc.so.6`___pthread_mutex_lock [inlined] lll_mutex_lock_optimized(mutex=0x00005555556e89e0) at pthread_mutex_lock.c:48:5
    frame #3: 0x00007ffff72a00f8 libc.so.6`___pthread_mutex_lock(mutex=0x00005555556e89e0) at pthread_mutex_lock.c:93:7
    frame #4: 0x00007ffff7571d59 libsofthsm2.so`OSLockMutex(mutex=0x00005555556e89e0) at osmutex.cpp:109:30
    frame #5: 0x00007ffff75719dd libsofthsm2.so`MutexFactory::LockMutex(this=0x00005555556b6970, mutex=0x00005555556e89e0) at MutexFactory.cpp:183:26
    frame #6: 0x00007ffff75715d1 libsofthsm2.so`Mutex::lock(this=0x00005555556e8710) at MutexFactory.cpp:61:50
    frame #7: 0x00007ffff7571669 libsofthsm2.so`MutexLocker::MutexLocker(this=0x00007fffffffc620, inMutex=0x00005555556e8710) at MutexFactory.cpp:82:32
    frame #8: 0x00007ffff75bf5b9 libsofthsm2.so`Token::isSOLoggedIn(this=0x00005555556e5300) at Token.cpp:95:29
    frame #9: 0x00007ffff75b9c28 libsofthsm2.so`SessionManager::openSession(this=0x00005555556e8320, slot=0x00005555556e4d50, flags=4, pApplication=0x00005555556e7df0, notify=(pkcs11.so`token_session_callback at session.c:44:1), phSession=0x00005555556e7e08) at SessionManager.cpp:83:58
    frame #10: 0x00007ffff7535af0 libsofthsm2.so`SoftHSM::C_OpenSession(this=0x00005555556d0c10, slotID=1146617904, flags=4, pApplication=0x00005555556e7df0, notify=(pkcs11.so`token_session_callback at session.c:44:1), phSession=0x00005555556e7e08) at SoftHSM.cpp:1414:40
    frame #11: 0x00007ffff750e07a libsofthsm2.so`C_OpenSession(slotID=1146617904, flags=4, pApplication=0x00005555556e7df0, notify=(pkcs11.so`token_session_callback at session.c:44:1), phSession=0x00005555556e7e08) at main.cpp:317:37
    frame #12: 0x00007ffff7dc2d69 pkcs11.so`p11prov_OpenSession(ctx=0x00005555556baa10, slotID=1146617904, flags=4, pApplication=0x00005555556e7df0, Notify=(pkcs11.so`token_session_callback at session.c:44:1), phSession=0x00005555556e7e08) at interface.gen.c:304:11
    frame #13: 0x00007ffff7de78e3 pkcs11.so`token_session_open(session=0x00005555556e7df0, flags=4) at session.c:76:19
    frame #14: 0x00007ffff7dea631 pkcs11.so`p11prov_get_session(provctx=0x00005555556baa10, slotid=0x00007fffffffc958, next_slotid=0x0000000000000000, uri=0x0000000000000000, mechtype=592, pw_cb=0x0000000000000000, pw_cbarg=0x0000000000000000, reqlogin=false, rw=false, _session=0x0000555555732470) at session.c:1023:19
    frame #15: 0x00007ffff7db41ca pkcs11.so`p11prov_digest_init(ctx=0x0000555555732460, params=0x0000000000000000) at digests.c:315:9
    frame #16: 0x00007ffff7846c9b libcrypto.so.3`evp_md_init_internal(ctx=0x0000555555728df0, type=0x0000555555731f00, params=0x0000000000000000, impl=0x0000000000000000) at digest.c:302:12
    frame #17: 0x00007ffff7846f8f libcrypto.so.3`EVP_DigestInit_ex(ctx=0x0000555555728df0, type=0x00007ffff7c8f8a0, impl=0x0000000000000000) at digest.c:382:12
    frame #18: 0x00007ffff75894ac libsofthsm2.so`OSSLEVPHashAlgorithm::hashInit(this=0x00005555556e8f10) at OSSLEVPHashAlgorithm.cpp:61:24
    frame #19: 0x00007ffff7596e48 libsofthsm2.so`RFC4880::PBEDeriveKey(password=0x00007fffffffcda0, salt=0x00007fffffffcb70, ppKey=0x00007fffffffcb58) at RFC4880.cpp:74:21
    frame #20: 0x00007ffff7597f7a libsofthsm2.so`SecureDataManager::login(this=0x00005555556e50e0, passphrase=0x00007fffffffcda0, encryptedKey=0x00005555556e50e8) at SecureDataManager.cpp:264:28
    frame #21: 0x00007ffff75983f5 libsofthsm2.so`SecureDataManager::loginUser(this=0x00005555556e50e0, userPIN=0x00007fffffffcda0) at SecureDataManager.cpp:317:30
    frame #22: 0x00007ffff75bfa28 libsofthsm2.so`Token::loginUser(this=0x00005555556e5300, pin=0x00007fffffffcda0) at Token.cpp:176:21
    frame #23: 0x00007ffff7535f53 libsofthsm2.so`SoftHSM::C_Login(this=0x00005555556d0c10, hSession=1, userType=1, pPin="12345678", ulPinLen=8) at SoftHSM.cpp:1538:25
    frame #24: 0x00007ffff750e328 libsofthsm2.so`C_Login(hSession=1, userType=1, pPin="12345678", ulPinLen=8) at main.cpp:407:31
    frame #25: 0x00007ffff7dc39ce pkcs11.so`p11prov_Login(ctx=0x00005555556baa10, hSession=1, userType=1, pPin="12345678", ulPinLen=8) at interface.gen.c:451:11
    frame #26: 0x00007ffff7de930a pkcs11.so`token_login(session=0x00005555556e7bb0, uri=0x0000000000000000, pw_cb=0x0000000000000000, pw_cbarg=0x0000000000000000, slot=0x0000555555712ec0, user_type=1) at session.c:565:11
    frame #27: 0x00007ffff7de9f3b pkcs11.so`slot_login(slot=0x0000555555712ec0, uri=0x0000000000000000, pw_cb=0x0000000000000000, pw_cbarg=0x0000000000000000, reqlogin=true, _session=0x0000000000000000) at session.c:834:15
    frame #28: 0x00007ffff7dea481 pkcs11.so`p11prov_get_session(provctx=0x00005555556baa10, slotid=0x00007fffffffd128, next_slotid=0x0000000000000000, uri=0x0000000000000000, mechtype=545, pw_cb=0x0000000000000000, pw_cbarg=0x0000000000000000, reqlogin=true, rw=false, _session=0x00005555556e6c90) at session.c:968:23
    frame #29: 0x00007ffff7dc07cf pkcs11.so`import_mac_key(macctx=0x00005555556e6c80, key="00112233445566778899aabbccddeeff", keylen=32, keyobj=0x00005555556e6c88) at mac.c:122:15
    frame #30: 0x00007ffff7dc0c3f pkcs11.so`p11prov_hmac_set_ctx_params(ctx=0x00005555556e6c80, params=0x00005555556e7d40) at mac.c:204:14
    frame #31: 0x00007ffff787c0ff libcrypto.so.3`EVP_MAC_CTX_set_params(ctx=0x00005555556ce2a0, params=0x00005555556e7d40) at mac_lib.c:220:16
    frame #32: 0x00005555555d7457 openssl`mac_main(argc=12, argv=0x00007fffffffd528) at mac.c:162:14
    frame #33: 0x00005555555dbe4a openssl`do_cmd(prog=0x00005555556b8860, argc=12, argv=0x00007fffffffd4d0) at openssl.c:428:16
    frame #34: 0x00005555555db9d2 openssl`main(argc=12, argv=0x00007fffffffd4d0) at openssl.c:309:19
    frame #35: 0x00007ffff722a1ca libc.so.6`__libc_start_call_main(main=(openssl`main at openssl.c:238:1), argc=13, argv=0x00007fffffffd4c8) at libc_start_call_main.h:58:16
    frame #36: 0x00007ffff722a28b libc.so.6`__libc_start_main_impl(main=(openssl`main at openssl.c:238:1), argc=13, argv=0x00007fffffffd4c8, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffd4b8) at libc-start.c:360:3
    frame #37: 0x00005555555a21a5 openssl`_start + 37

The mutex is locked during Token::loginUser() (frame #22) and is still locked when Token::isSOLoggedIn() is called (frame #8).

I got the test working without early loading or forcing all operations on the token, it was just a matter of initializing pkcs11 a bit earlier and adding HMAC to static_operations_init().

[simo5]

softhsm is hopelessly broken like that in many places, feel free to exclude the test on softhsm

[simo5]

Note that it is likely you will never be able to use softhsm with this feature, if that is your target token you may want to consider switching to use kryoptic as token instead (there is a utility to export existing softhsm databases to its format too).

[ilie-halip-nxp]

My main target is NXP's SMW library (https://github.com/nxp-imx/imx-smw) which exposes a PKCS11 interface to a secure enclave that can do TLS derivation without exporting the key material, thus the need for the HMAC operation as in this PR.

If I do all operations on the token in the test as you mentioned, wouldn't both openssl mac commands do exactly the same thing?

[simo5]

When you say "both" what do you mean?

[ilie-halip-nxp]

The test script, thmac, runs openssl mac 1) with and 2) without -propquuery ?provider=pkcs11, then compares the outputs.

[simo5]

the results should always be the same, certainly

[ilie-halip-nxp]

Yes, and I saw in other tests that running all operations on the token implies setting
default_properties = ?provider=pkcs11
in openssl.cnf.

In this case, both openssl mac command would invoke the pkcs11 provider implementation. Is this right?

[ilie-halip-nxp]

Early loading seem odd to need, can you instead force all operations on the token?

I may have misunderstood what you meant by this, could you take a look at the last commit(s) in this series of fixups? I'm not sure I went the right way with my changes. Many thanks!

[Jakuje]

Can you rebase on top of the current master? Rebuild should fix the centos10 issues. Then we are left with the softhsm issues.

[ilie-halip-nxp]

I'm going to close this PR as it's become stale. I will open a new one after rebasing/squashing and a couple more updates.