Skip to content

Support different OAEP modes using OpenSSL EVP interface #583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
borancar wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Support different OAEP modes using OpenSSL EVP interface #583

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
abaa07e
Plumb in different hash options for RSA_PKCS_OAEP
borancar Oct 29, 2020
357bacb
Implement OSSLRSA ops using EVP interface
borancar Oct 29, 2020
f2a92a3
Check that MGF1 matches hash alg for OAEP
borancar Oct 30, 2020
fe08df8
Add OAEP tests and fix padding check
borancar Oct 30, 2020
6ecd7ec
Bump OpenSSL version
borancar Oct 30, 2020
69b801e
Explain else branch for openssl
borancar Jan 25, 2021
8498fa2
Fix typo in error message
borancar Sep 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ SoftHSM depends on a cryptographic library, Botan or OpenSSL.
Minimum required versions:

- Botan 2.0.0
- OpenSSL 1.0.0
- OpenSSL 1.0.2

If you are using Botan, use at least version 2.6.0. This will improve
the performance when doing public key operations.
Expand Down
6 changes: 4 additions & 2 deletions m4/acx_crypto_backend.m4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,11 @@ AC_DEFUN([ACX_CRYPTO_BACKEND],[
AC_MSG_RESULT(OpenSSL)

if test "x${enable_fips}" = "xyes"; then
ACX_OPENSSL(1,0,1)
# needed for FIPS compliance, so change only when FIPS requirements change
ACX_OPENSSL(1,0,2)
else
ACX_OPENSSL(1,0,0)
# increase this as features from newer versions needed
ACX_OPENSSL(1,0,2)
fi

CRYPTO_INCLUDES=$OPENSSL_INCLUDES
Expand Down
168 changes: 149 additions & 19 deletions src/lib/SoftHSM.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2373,6 +2373,7 @@ CK_RV SoftHSM::AsymEncryptInit(CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMec

// Get the asymmetric algorithm matching the mechanism
AsymMech::Type mechanism;
unsigned long expectedMgf;
bool isRSA = false;
switch(pMechanism->mechanism) {
case CKM_RSA_PKCS:
Expand All @@ -2394,7 +2395,37 @@ CK_RV SoftHSM::AsymEncryptInit(CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMec
if (rv != CKR_OK)
return rv;

mechanism = AsymMech::RSA_PKCS_OAEP;
switch(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->hashAlg) {
case CKM_SHA_1:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA1;
expectedMgf = CKG_MGF1_SHA1;
break;
case CKM_SHA224:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA224;
expectedMgf = CKG_MGF1_SHA224;
break;
case CKM_SHA256:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA256;
expectedMgf = CKG_MGF1_SHA256;
break;
case CKM_SHA384:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA384;
expectedMgf = CKG_MGF1_SHA384;
break;
case CKM_SHA512:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA512;
expectedMgf = CKG_MGF1_SHA512;
break;
default:
DEBUG_MSG("hashAlg must be one of: CKM_SHA_1, CKM_SHA224, CKM_SHA256, CKM_SHA384, CKM_SHA512");
return CKR_ARGUMENTS_BAD;
}

if(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->mgf != expectedMgf) {
ERROR_MSG("Hash and MGF don't match");
return CKR_ARGUMENTS_BAD;
}

isRSA = true;
break;
default:
Expand Down Expand Up @@ -3097,6 +3128,7 @@ CK_RV SoftHSM::AsymDecryptInit(CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMec

// Get the asymmetric algorithm matching the mechanism
AsymMech::Type mechanism = AsymMech::Unknown;
unsigned long expectedMgf;
bool isRSA = false;
switch(pMechanism->mechanism) {
case CKM_RSA_PKCS:
Expand All @@ -3120,18 +3152,38 @@ CK_RV SoftHSM::AsymDecryptInit(CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMec
DEBUG_MSG("pParameter must be of type CK_RSA_PKCS_OAEP_PARAMS");
return CKR_ARGUMENTS_BAD;
}
if (CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->hashAlg != CKM_SHA_1)
{
DEBUG_MSG("hashAlg must be CKM_SHA_1");
return CKR_ARGUMENTS_BAD;

switch(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->hashAlg) {
case CKM_SHA_1:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA1;
expectedMgf = CKG_MGF1_SHA1;
break;
case CKM_SHA224:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA224;
expectedMgf = CKG_MGF1_SHA224;
break;
case CKM_SHA256:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA256;
expectedMgf = CKG_MGF1_SHA256;
break;
case CKM_SHA384:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA384;
expectedMgf = CKG_MGF1_SHA384;
break;
case CKM_SHA512:
mechanism = AsymMech::RSA_PKCS_OAEP_SHA512;
expectedMgf = CKG_MGF1_SHA512;
break;
default:
DEBUG_MSG("hashAlg must be one of: CKM_SHA_1, CKM_SHA224, CKM_SHA256, CKM_SHA384, CKM_SHA512");
return CKR_ARGUMENTS_BAD;
}
if (CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->mgf != CKG_MGF1_SHA1)
{
DEBUG_MSG("mgf must be CKG_MGF1_SHA1");

if (CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->mgf != expectedMgf) {
ERROR_MSG("Hash and MGF don't match");
return CKR_ARGUMENTS_BAD;
}

mechanism = AsymMech::RSA_PKCS_OAEP;
isRSA = true;
break;
default:
Expand Down Expand Up @@ -6221,6 +6273,7 @@ CK_RV SoftHSM::WrapKeyAsym
const size_t bb = 8;
AsymAlgo::Type algo = AsymAlgo::Unknown;
AsymMech::Type mech = AsymMech::Unknown;
unsigned long expectedMgf;

CK_ULONG modulus_length;
switch(pMechanism->mechanism) {
Expand All @@ -6247,11 +6300,51 @@ CK_RV SoftHSM::WrapKeyAsym
break;

case CKM_RSA_PKCS_OAEP:
mech = AsymMech::RSA_PKCS_OAEP;
// SHA-1 is the only supported option
// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 160 / 8)
return CKR_KEY_SIZE_RANGE;
switch(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->hashAlg) {
case CKM_SHA_1:
mech = AsymMech::RSA_PKCS_OAEP_SHA1;
expectedMgf = CKG_MGF1_SHA1;
// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 160 / 8)
return CKR_KEY_SIZE_RANGE;
break;
case CKM_SHA224:
mech = AsymMech::RSA_PKCS_OAEP_SHA224;
expectedMgf = CKG_MGF1_SHA1;
Copy link
Contributor

@antoinelochet antoinelochet Mar 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This (and the other expectedMgf below for the other hash functions) is not the same as the other expectedMgf is the modification above.
It seems weird.
I think it should be CKG_MGF1_SHA224 which would make more sense.

Copy link
Author

@borancar borancar Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this @antoinelochet?

// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 224 / 8)
return CKR_KEY_SIZE_RANGE;
break;
case CKM_SHA256:
mech = AsymMech::RSA_PKCS_OAEP_SHA256;
expectedMgf = CKG_MGF1_SHA1;
// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 256 / 8)
return CKR_KEY_SIZE_RANGE;
break;
case CKM_SHA384:
mech = AsymMech::RSA_PKCS_OAEP_SHA384;
expectedMgf = CKG_MGF1_SHA1;
// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 384 / 8)
return CKR_KEY_SIZE_RANGE;
break;
case CKM_SHA512:
mech = AsymMech::RSA_PKCS_OAEP_SHA512;
expectedMgf = CKG_MGF1_SHA1;
// PKCS#11 2.40 draft 2 section 2.1.8: input length <= k-2-2hashLen
if (keydata.size() > modulus_length - 2 - 2 * 512 / 8)
return CKR_KEY_SIZE_RANGE;
break;
default:
return CKR_MECHANISM_INVALID;
}

if(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->mgf != expectedMgf) {
ERROR_MSG("Hash and MGF don't match");
return CKR_ARGUMENTS_BAD;
}

break;

default:
Expand Down Expand Up @@ -6629,6 +6722,7 @@ CK_RV SoftHSM::UnwrapKeyAsym
// Get the symmetric algorithm matching the mechanism
AsymAlgo::Type algo = AsymAlgo::Unknown;
AsymMech::Type mode = AsymMech::Unknown;
unsigned long expectedMgf;
switch(pMechanism->mechanism) {
case CKM_RSA_PKCS:
algo = AsymAlgo::RSA;
Expand All @@ -6637,7 +6731,35 @@ CK_RV SoftHSM::UnwrapKeyAsym

case CKM_RSA_PKCS_OAEP:
algo = AsymAlgo::RSA;
mode = AsymMech::RSA_PKCS_OAEP;
switch(CK_RSA_PKCS_OAEP_PARAMS_PTR(pMechanism->pParameter)->hashAlg) {
case CKM_SHA_1:
mode = AsymMech::RSA_PKCS_OAEP_SHA1;
expectedMgf = CKG_MGF1_SHA1;
break;
case CKM_SHA224:
mode = AsymMech::RSA_PKCS_OAEP_SHA224;
expectedMgf = CKG_MGF1_SHA224;
break;
case CKM_SHA256:
mode = AsymMech::RSA_PKCS_OAEP_SHA256;
expectedMgf = CKG_MGF1_SHA256;
break;
case CKM_SHA384:
mode = AsymMech::RSA_PKCS_OAEP_SHA384;
expectedMgf = CKG_MGF1_SHA384;
break;
case CKM_SHA512:
mode = AsymMech::RSA_PKCS_OAEP_SHA512;
expectedMgf = CKG_MGF1_SHA512;
break;
default:
return CKR_MECHANISM_INVALID;
}

if (CK_RSA_PKCS_PSS_PARAMS_PTR(pMechanism->pParameter)->mgf != expectedMgf) {
ERROR_MSG("Hash and MGF don't match");
return CKR_ARGUMENTS_BAD;
}
break;

default:
Expand Down Expand Up @@ -12392,14 +12514,22 @@ CK_RV SoftHSM::MechParamCheckRSAPKCSOAEP(CK_MECHANISM_PTR pMechanism)
}

CK_RSA_PKCS_OAEP_PARAMS_PTR params = (CK_RSA_PKCS_OAEP_PARAMS_PTR)pMechanism->pParameter;
if (params->hashAlg != CKM_SHA_1)
if (params->hashAlg != CKM_SHA_1 &&
params->hashAlg != CKM_SHA224 &&
params->hashAlg != CKM_SHA256 &&
params->hashAlg != CKM_SHA384 &&
params->hashAlg != CKM_SHA512)
{
ERROR_MSG("hashAlg must be CKM_SHA_1");
ERROR_MSG("hashAlg must be one of: CKM_SHA_1, CKM_SHA224, CKM_SHA256, CKM_SHA384, CKM_SHA512");
return CKR_ARGUMENTS_BAD;
}
if (params->mgf != CKG_MGF1_SHA1)
if (params->mgf != CKG_MGF1_SHA1 &&
params->mgf != CKG_MGF1_SHA224 &&
params->mgf != CKG_MGF1_SHA256 &&
params->mgf != CKG_MGF1_SHA384 &&
params->mgf != CKG_MGF1_SHA512)
{
ERROR_MSG("mgf must be CKG_MGF1_SHA1");
ERROR_MSG("mgf must be one of: CKG_MGF1_SHA1, CKM_MGF1_SHA224, CKM_MGF1_SHA256, CKM_MGF1_SHA384, CKM_MGF1_SHA512");
return CKR_ARGUMENTS_BAD;
}
if (params->source != CKZ_DATA_SPECIFIED)
Expand Down
6 changes: 5 additions & 1 deletion src/lib/crypto/AsymmetricAlgorithm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,11 @@ bool AsymmetricAlgorithm::isWrappingMech(AsymMech::Type padding)
{
case AsymMech::RSA:
case AsymMech::RSA_PKCS:
case AsymMech::RSA_PKCS_OAEP:
case AsymMech::RSA_PKCS_OAEP_SHA1:
case AsymMech::RSA_PKCS_OAEP_SHA224:
case AsymMech::RSA_PKCS_OAEP_SHA256:
case AsymMech::RSA_PKCS_OAEP_SHA384:
case AsymMech::RSA_PKCS_OAEP_SHA512:
return true;

default:
Expand Down
6 changes: 5 additions & 1 deletion src/lib/crypto/AsymmetricAlgorithm.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,11 @@ struct AsymMech
RSA,
RSA_MD5_PKCS,
RSA_PKCS,
RSA_PKCS_OAEP,
RSA_PKCS_OAEP_SHA1,
RSA_PKCS_OAEP_SHA224,
RSA_PKCS_OAEP_SHA256,
RSA_PKCS_OAEP_SHA384,
RSA_PKCS_OAEP_SHA512,
RSA_SHA1_PKCS,
Comment on lines +68 to 73
Copy link

@coderabbitai coderabbitai bot Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Enum split to per-hash OAEP variants. Verify no lingering uses of the old value.

Removing RSA_PKCS_OAEP in favor of explicit SHA* variants is fine. Please ensure no references to the removed enum remain (including Botan backend paths).

Run:

🏁 Script executed:

#!/bin/bash
# Find any stale references to the old enum and confirm both backends handle new ones.
rg -nP --type=cpp -C2 '\bRSA_PKCS_OAEP\b|RSA_PKCS_OAEP_SHA(1|224|256|384|512)' src/

# Sanity check OpenSSL and Botan RSA implementations for new cases.
fd -a 'OSSLRSA.cpp' src/ | xargs -I{} rg -n 'RSA_PKCS_OAEP_SHA' {}
fd -a 'BotanRSA.cpp' src/ | xargs -I{} rg -n 'RSA_PKCS_OAEP_SHA|OAEP' {}

Length of output: 13233

Leftover AsymMech::RSA_PKCS_OAEP references remain — fix Botan and check MGF mappings.

Botan backend still uses AsymMech::RSA_PKCS_OAEP; change to AsymMech::RSA_PKCS_OAEP_SHA1 (Botan uses EME1(SHA-160)). See src/lib/crypto/BotanRSA.cpp:764 and src/lib/crypto/BotanRSA.cpp:843.
Also review src/lib/SoftHSM.cpp (approx. lines 6311–6335): several cases set expectedMgf = CKG_MGF1_SHA1 for non‑SHA1 hashes — confirm or correct.

RSA_SHA224_PKCS,
RSA_SHA256_PKCS,
Expand Down
Loading
Loading