RSA signature verification API vulnerability in the r_sce_protected module

[Kei-Matsuoka-jpn]

Issue

In SCE Protected Mode (r_sce_protected), the R_SCE_RSASSA_PKCSxxxx_SignatureVerify() APIs can trigger a pointer underflow when the provided signature is larger than the expected size (signature->data_length > HW_SCE_RSA_xxxx_DATA_BYTE_SIZE, where xxxx is 1024, 2048, 3072, or 4096).

The underflow occurs because the API copies signature->data_length bytes into a fixed-size buffer using an offset calculated as:

ptr_data = HW_SCE_RSA_xxxx_DATA_BYTE_SIZE - signature->data_length

If signature->data_length exceeds the buffer size, ptr_data becomes negative, leading to out-of-bounds writes and potential data corruption.

Workaround

Before calling the API, ensure that:

signature->data_length <= HW_SCE_RSA_xxxx_DATA_BYTE_SIZE

This prevents the pointer underflow and memory corruption.

Acknowledgement:

Renesas would like to thank SecMate for their responsible reporting of their findings.