i2d_PublicKey() fails with -1 for DILITHIUM2 key while using OQS provider with OpenSSL 3.2.1

[vk18anubhav]

Hi Team,

I am generating a DILITHIUM2 key in my code using OpenSSL 3.2.1 by loading OQS provider as following

EVP_PKEY *pkey = NULL;
EVP_PKEY_CTX *ctx = (EVP_PKEY_CTX *)NULL;

pkey = EVP_PKEY_new();

ctx = EVP_PKEY_CTX_new_from_name(libctx, "dilithium2", NULL);

EVP_PKEY_keygen_init(ctx);

EVP_PKEY_keygen(ctx, &pkey);

While converting the key to DER format using i2d_PublicKey() , it fails with -1 while i2d_PrivateKey() succeeds.

len = i2d_PublicKey(pkey, NULL); // First call itself fails

if(len < buffer->len)
buffer->len = i2d_PublicKey(pkey, buffer); // This also fails 

I tried debugging the issue by putting breakpoints on i2d_PublicKey(), i2d_PrivateKey() and sharing observations

int i2d_PublicKey(const EVP_PKEY *a, unsigned char **pp)
{
    if (evp_pkey_is_provided(a)) {
        static const struct type_and_structure_st output_info[] = {
            { "DER", "type-specific" },
            { "blob", NULL },    /* for EC */
            { NULL, }
        };
 
        return i2d_provided(a, EVP_PKEY_PUBLIC_KEY, output_info, pp);
    }

int i2d_PrivateKey(const EVP_PKEY *a, unsigned char **pp)
{
    if (evp_pkey_is_provided(a)) {
        static const struct type_and_structure_st output_info[] = {
            { "DER", "type-specific" },
            { "DER", "PrivateKeyInfo" },
            { NULL, }
        };
 
        return i2d_provided(a, EVP_PKEY_KEYPAIR, output_info, pp);
    }

• i2d_provided() calls OSSL_ENCODER_CTX_new_for_pkey(), OSSL_ENCODER_to_data() in loop for members of output_info
• OSSL_ENCODER_CTX_new_for_pkey () was using oqsprovider for getting libctx
• OSSL_ENCODER_to_data() returns doing nothing for output_type= DER, output_structure= "type-specific" for both private and public key.
• OSSL_ENCODER_to_data() succeeds for next output_type= DER, output_structure= “PrivateKeyInfo" while it fails for output_type= blob, output_structure= NULL for public key and returns with -1

Can you please review on how I can resolve this ?