Performance degradation, no_store/no_cache flag set

[woinan]

Hi all

I've being benchmarking OpenSSL v3.4.x with oqs_provider 0.10.0 + liboqs 0.14.0 and observed a performance issue/drop which seems to be related to no_store/no_cache flag set. (

oqs-provider/oqsprov/oqsprov.c

Line 976 in 7200b2c

*no_cache = 1;

).

Could you please tell me is it safe to revert that change (*no_cache = 1;) and re-enable the fetch caching ?

Thank you!

[baentsch]

Could you please tell me is it safe to revert that change (*no_cache = 1;) and re-enable the fetch caching ?

At first glance and in general, this does not seem advisable: The reason for disabling caching is that the set of supported algorithms can change during run time -- enabling which was the very purpose of 1246c33. Therefore, allowing caching would effectively make run-time changes to the set of algorithms effectively impossible (as initially reported algorithm set and run-time query set diverge on subsequent calls to the _query function). That said, if you only use OpenSSL 3.4 you should be safe doing that as runtime-algorithm changes should not happen in that version. But relying on a feature that is version-dependent also doesn't seem advisable :). And out of curiosity: Which degree of performance degradation do you see @woinan ?

[woinan]

Thank you @baentsch. Nginx 1C2T PQC RPS benchmark for mldsa44-mlkem512 shows 4x higher RPS when caching is enabled.

[baentsch]

Thank you @baentsch. Nginx 1C2T PQC RPS benchmark for mldsa44-mlkem512 shows 4x higher RPS when caching is enabled.

Wow - that's massive! Can you point to which benchmark you're referring-to exactly? I'd like to reproduce this -- and maybe do a PR to disable it when running in an OpenSSL < 3.5 installation then -- unless of course, you'd like to contribute that @woinan (?). Maybe also interesting is a comparison between OpenSSL3.4+oqsprovider with OpenSSL3.5 -- or did you also already do that?

[woinan]

I run RPS PQC benchmarks of Nginx with OpenSSL v3.4.2 (PQC via oqs_provider v0.10.0 and IA optimized liboqs v0.14.0) and OpenSSL v3.5.0 (native PQC).

In the test setup, Nginx (https://github.com/intel/asynch_mode_nginx) is running on two logical threads of single IA core, Nginx is configured to respond with "HTTP 204 No Content" (~100Bytes) to GET "/204" request.
On a client side I run 8-16 instances of (a bit modified version of) WRK, 1 thread and 64 connections per instance.

While benchmarking Nginx and OpenSSL v3.4.2 with AI optimized liboqs, I’ve observed performance somehow comparable (+/- 30%) to OpenSSL v3.5.0 with native, unoptimized PQC.
That was rather unexpected as openssl speed results for OpenSSL v3.4.2 with AI optimized liboqs were much higher (e.g.: ~3-4x higher for mlkem512).

Analysis of FlameGraphs (attached) led to conclusion that provider’s algorithm fetches are not being cached.
With caching enabled the PQC RPS OpenSSL v3.4.2 with AI optimized liboqs results are 3-4.5x higher.

Working on the contribution.

nginx.conf
nginx-mlkem512-mldsa44.conf.sanitized.txt

v3.4.2

v3.4.2 caching enabled

v3.5.0

[tkanteck]

@baentsch
Since there is a performance gap between openssl 3.5+ and liboqs with respect to ML-DSA and ML-KEM on x86, and no_cache option has major impact on application performance, the oqsprovider could allow users to control these behaviors. The default behavior should remain as it is today.
I propose introducing two parameters in the provider logic:

1. option that enables PQC algos with openssl 3.5+ (default: disable PQC algorithms.)
2. option to control no_cache logic (default behavior: do not cache). BTW are there any known applications that change providers at runtime?

If this sounds reasonable, we can draft a PR with the above changes.

[baentsch]

Thanks for these proposals, @tkanteck . PRs are always welcome. I'm just not sure what

option that enables PQC algos with openssl 3.5+ (default: disable PQC algorithms.)

means? Can I assume you mean "option that enables NIST standardized PQC algos with 3.5+ for use with liboqs-based implementations"? If so, that's not quite easy. But again, if you think there's an easy way, by all means, please make a PR proposal.

BTW are there any known applications that change providers at runtime?

That I don't know. The curse of the library developer: One never knows how the stuff's getting used. Secondly, I'm not sure what this possibility would have to do with provider-internal caching behaviour: Could you please clarify?

[tkanteck]

Thanks @baentsch. Let's tackle things one by one and focus on no_cache option that affects application performance with openssl < 3.5.

The curse of the library developer: One never knows how the stuff's getting used. Secondly, I'm not sure what this possibility would have to do with provider-internal caching behaviour: Could you please clarify?

I completely understand that feeling! 🙂
Would you be open to introducing a mechanism - such as an environment variable (OQS_NO_CACHE) or a new configuration parameter - to control the no_cache setting instead of hardcoding it to 1?

Hardcoding it to 1 every time the oqsprovider_query function is called, means OpenSSL will not cache responses for operation queries, forcing the provider code to be queried each time. As observed on the flame graphs above, CPU spends significant amount of time searching for the method which degrades application performance.

no_cache control mechanism would improve application level performance and make the library more appealing to use cases where changing providers at run-time is not needed.

[tkanteck]

Does it sound reasonable? If there is no objection we'll draft a PR for this functionality. Thanks

[baentsch]

Does it sound reasonable? If there is no objection we'll draft a PR for this functionality. Thanks

I'm not convinced that giving users the ability to disable a functionality that has been added for a reason is the best way forward: This calls for creating support issues ("why does your code fail under situation xyz?"). Anyway, I may not fully grasp your suggestion, so please feel free to put forward a draft PR so I better understand it. Maybe the actual code would alleviate my theoretical concerns :)

[woinan]

@baentsch please see PR. Thank you!

[baentsch]

@baentsch please see PR. Thank you!

Err -- a bit unusual to check a fork -- but OK, let me see whether I can comment there too...

[woinan]

Thank you for review! Please see updated PR.

[woinan]

@baentsch please see updated PR. Thank you!