Skip to content

Conversation

@andrewyounkers
Copy link

This change looks to address a crash reported by Roumen Petrov (pkixssh@roumenpetrov.info), the developer of PKIX-SSH. The vulnerability exists in the error handling paths of kexoqsecdh.c and kexoqsx25519.c in the OQS_KEM_encaps() or OQS_KEM_decaps() functions. When OQS key encapsulation operations fail, the return value does not indicate a failure. On the error path the code uses the value 0 for variable r set from previous operations, causing the code to try to use either server_blobp or shared_secretp causing a NULL => bus error and crash.

In kexoqs.c, error handing is handled before jumping to the cleanup code in the OQS_KEM_encaps and OQS_KEM_decaps functions by setting r = SSH_ERR_LIBCRYPTO_ERROR;. This PR makes a change to have the error handling logic behave the same way in kexoqsecdh.c and kexoqsx25519.c.

I was able to reproduce this crash using the PKIX-SSH client and the OQS-SSH server when KEX algorithm list truncation caused a mismatch and failure by triggering a MAX_PROP buffer limitation in the client's match.c file. The confirmation was seeing mm_reap: preauth child terminated by signal 11 in the server logs, and then verifying that line was no longer present after this change.

The reproduction steps/attack scenario this addresses are as follows:

  1. A client successfully negotiates a hybrid post-quantum KEX algorithm
  2. The client sends bad KEX data
  3. The server's OQS_KEM_encaps() or OQS_KEM_decaps() fails to process the bad data
  4. The function returns r=0 despite the failure
  5. Output pointers (*server_blobp, *shared_secretp) remain NULL
  6. The caller attempts to dereference these NULL pointers
  7. Server crashes

Signed-off-by: Andrew Younkers <ayounkers44@gmail.com>
Signed-off-by: Andrew Younkers <ayounkers44@gmail.com>
@dstebila dstebila self-requested a review December 16, 2025 17:49
@baentsch baentsch requested a review from a team December 16, 2025 17:50
@dstebila dstebila merged commit 8b14bb4 into open-quantum-safe:OQS-v10 Jan 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants