EVP_VerifyFinal() failed: invalid padding

[TobiMarg]

I am trying to setup pam_pkcs11 using an OpenPGP card. Everything seems to work except the signature verification. Here is a snipped from the output:

DEBUG:pam_pkcs11.c:618: certificate is valid and matches the user
Checking signature
DEBUG:pkcs11_lib.c:139: reading 128 random bytes from /dev/urandom
DEBUG:pkcs11_lib.c:157: random-value[128] = [b3:88:65:...:c6]
DEBUG:pkcs11_lib.c:1735: private key type: 0x00000000
DEBUG:pkcs11_lib.c:1807: hash[51] = [...:9c:83:d0:...:e4]
DEBUG:pkcs11_lib.c:1856: signature[512] = [81:86:05:...:2e]
DEBUG:pam_pkcs11.c:681: verifying signature...
DEBUG:cert_vfy.c:533: public key type: 0x00000006
DEBUG:cert_vfy.c:534: public key bits: 0x00001000
DEBUG:cert_vfy.c:566: hashing with SHA256
ERROR:pam_pkcs11.c:688: verify_signature() failed: EVP_VerifyFinal() failed: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding
Error 2342: Verifying signature failed

Simply signing and verifying some data with pkcs11-tool like this is successful:

pkcs11-tool --sign -i testdata --id 03 --output-file testdata.sig_p11
pkcs11-tool --verify -i testdata --signature-file testdata.sig_p11 --id 03

I compiled pam_pkcs11 from sources today, hence this seems to be a bug in the current version. If there is anything I can help to debug this I am willing to try. :)

[frankmorgner]

please try to restrict pkcs11-tool to CKM_RSA_PKCS (--mechanism), and then use openssl rather than pkcs11-tool to verify this with the certificate from the card.