Compatibility layer additions for X.509 extensions and RSA PSS

[anhu]

• Fix for compat wolfSSL_RSA_sign and wolfSSL_RSA_verify to support RSA PSS with custom salt and mgf1 hash type.
• Adds compat API's for i2d_PrivateKey_bio , BN_ucmp and X509v3_get_ext_by_NID.
  ZD 20059

[anhu]

These features expect the following configuration:
./configure --enable-opensslall --enable-keygen

[anhu]

Jenkins retest this please.

Cannot contact wolf-linux-cloud-node-tosotv: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 636F7272

[anhu]

Jenkins retest this please

Found unhandled hudson.remoting.RequestAbortedException exception:
java.io.StreamCorruptedException: invalid stream header: 636F7272
	hudson.remoting.Request.abort(Request.java:358)
	hudson.remoting.Channel.terminate(Channel.java:[1](https://cloud.wolfssl-test.com/jenkins/job/wolfSSL/job/PRB-140-3-tests/5574/pipeline-console/?start-byte=0&selected-node=207#log-1)196)
	hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:95)

[anhu]

Jenkins retest this please

Found unhandled org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException exception:

[anhu]

Regarding WC_RSA_PSS . It is insufficient. We would also need WC_RSA_NO_PADDING. opensslall brings in both I think. Not sure its worth it to go so fine grained.

[anhu]

@dgarske let me know your opinion. I can either revert the suggestions or move more fine grained.

[dgarske]

@anhu please review the errors. Looks like this needs a little more work.

[anhu]

jenkins retest this please

Found unhandled hudson.remoting.RequestAbortedException exception:
java.io.StreamCorruptedException: invalid stream header: 636F7272

[anhu]

I cannot gate on WC_RSA_PSS. My changes call wolfSSL_RSA_verify_PKCS1_PSS_mgf1() and wolfSSL_RSA_padding_add_PKCS1_PSS_mgf1() which are gated by OPENSSLALL

[dgarske]

Code is good now (in my opinion), but it will need squash before merge.

[Copilot]

Pull Request Overview

Adds compatibility support for RSA-PSS with custom salt lengths and MGF1 hashing, plus utility APIs for private key BIO output, unsigned BIGNUM comparison, and X.509 extension lookup.

• Introduces wolfSSL_i2d_PrivateKey_bio and maps it to the OpenSSL alias.
• Extends EVP_PKEY_CTX with PSS salt length and MGF1-Hash setters; updates RSA sign/verify to use these parameters.
• Adds wolfSSL_BN_ucmp and X509v3 extension lookup functions (wolfSSL_X509v3_get_ext*), with corresponding OpenSSL-compatible macros.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
wolfssl/ssl.h	Declared wolfSSL_i2d_PrivateKey_bio
wolfssl/openssl/x509v3.h	Added wolfSSL_X509v3_get_ext and wolfSSL_X509v3_get_ext_by_NID
wolfssl/openssl/ssl.h	Mapped i2d_PrivateKey_bio to wolfSSL_i2d_PrivateKey_bio
wolfssl/openssl/rsa.h	Defined RSA-PSS salt-length constants and alias macros
wolfssl/openssl/evp.h	Added PSS saltlen and MGF1-hash setter prototypes
wolfssl/openssl/bn.h	Introduced wolfSSL_BN_ucmp and alias macro
wolfcrypt/src/rsa.c	Reformatted multi-line #if for direct RSA
wolfcrypt/src/evp.c	Initialized new fields in WOLFSSL_EVP_PKEY_CTX and implemented setters and integration for PSS parameters
tests/api.c	Added tests for RSA-PSS sign/verify and X509v3 extension stack lookup
src/x509.c	Implemented wolfSSL_X509v3_get_ext* functions
src/ssl_bn.c	Implemented unsigned BIGNUM comparison (BN_ucmp)
src/ssl.c	Implemented wolfSSL_i2d_PrivateKey_bio
src/pk.c	Refactored PSS salt-length calculation into helper function

Comments suppressed due to low confidence (3)

src/ssl_bn.c:1162

• The new function wolfSSL_BN_ucmp lacks corresponding unit tests to verify its correctness across various cases (e.g., comparing positive, negative, and NULL BIGNUMs). Consider adding tests in tests/api.c to cover these scenarios.

int wolfSSL_BN_ucmp(const WOLFSSL_BIGNUM* a, const WOLFSSL_BIGNUM* b)

wolfssl/openssl/evp.h:952

• [nitpick] Consider adding documentation comments (e.g., Doxygen) for this new API to clarify its behavior, valid salt length values, and error conditions, improving maintainability and discoverability.

WOLFSSL_API int wolfSSL_EVP_PKEY_CTX_set_rsa_pss_saltlen(WOLFSSL_EVP_PKEY_CTX *ctx, int saltlen);

tests/api.c:5143

• [nitpick] The test function name uses a wc_ prefix but other OpenSSL-compat tests use test_wolfSSL_ prefixes (e.g., test_wolfSSL_X509_*); consider renaming this for consistency.

static int test_wc_RsaPSS_DigitalSignVerify(void)

[dgarske]

Jenkins retest this please: "Found unhandled org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException exception:"

[anhu]

Jenkins retest this please

Oops! Not found.

[dgarske]

Jenkins retest this please. The FIPS-repo-and-harness-part1 test is failing, but history keeps being removed too soon.

[anhu]

Jenkins retest this please.

Ooops! Not found.

[anhu]

Running config: --enable-fips=v2 --enable-opensslextra...
Config Good
make[2]: warning: -j3 forced in submake: resetting jobserver mode.
In file included from src/ssl.c:360:
./src/pk.c:3540:12: error: ‘rsa_pss_calc_salt’ defined but not used [-Werror=unused-function]
3540 | static int rsa_pss_calc_salt(int saltLen, int hashLen, int emLen)
| ^~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors

[anhu]

Customer has approved these changes.

[anhu]

Jenkins retest this please.

Found unhandled org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException exception:
Unable to create live FilePath for wolf-linux-cloud-node-letr41; wolf-linux-cloud-node-letr41 was marked offline: Connection was broken