Add cmake "run" target and use it in PR CI #53

esyr · 2025-10-16T16:41:16Z

Introduce a run target, that executes all the test with different options. For now is used only for minimal sanity checking, that are test can be run and don't fail, but later can be extended to check the results for sanity as well, and the design has some provisions put in place to be useful for test data collection as well, for example.

jogme

lgtm, thank you!

quarckster · 2025-10-17T08:42:17Z

Interesting that the only Windows on master didn't pass.

npajkovsky · 2025-10-17T08:43:30Z

source/CMakeLists.txt

+            endforeach()
+            set(cmds ${new_cmds})
+        endif()
+    endforeach()


I have tried it but make run does not respect or set (DY)LD_LIBRARY_PATH which should be fixed.

I would probably write custom target manually for each binary. The cmd options are much too different for a generic solution.

$ echo $DYLD_LIBRARY_PATH /Users/npajkovsky/openssl/openssl $ ll /Users/npajkovsky/openssl/openssl | grep ssl.4 -rwxr-xr-x 1 npajkovsky staff 960K Oct 17 10:21 libssl.4.dylib $ make run [ 2%] Built target perf [ 3%] Built target x509storeissuer [ 3%] Run x509storeissuer -t /Users/npajkovsky/openssl/openssl/test/certs 4 dyld[58574]: Library not loaded: /usr/local/lib/libssl.4.dylib Referenced from: <7C191CC0-3566-3B84-9EB4-9AB01FD44703> /Users/npajkovsky/openssl/perftools/source/x509storeissuer Reason: tried: '/usr/local/lib/libssl.4.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libssl.4.dylib' (no such file), '/usr/local/lib/libssl.4.dylib' (no such file) make[3]: *** [CMakeFiles/run-x509storeissuer--t--U_sers-npajkovsky-openssl-openssl-test-certs-4] Abort trap: 6 make[2]: *** [CMakeFiles/run-x509storeissuer--t--U_sers-npajkovsky-openssl-openssl-test-certs-4.dir/all] Error 2 make[1]: *** [CMakeFiles/run.dir/rule] Error 2 make: *** [run] Error 2

Have you exported the variable? But that's a valid point, that cmake doesn't do any provisions to supply the provided OPENSSL_ROOT_DIR to the executables being run, the relevant overrides should be added to the CI recipes.

Works fine on OpenBSD where I run cmake as follows:

cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install cmake --build build cmake --build build -t run

The build is setup such linker adds RUNPATH to elf header. The RUNPATH is set to /path/to/openssl.install I will take a look at macos.

In my case the failure happens later due to missing/wrong path to testcert dirs.

[ 42%] Built target run-evp-setpeer--k-ec256--t-4 [ 42%] Built target handshake [ 42%] Run handshake -P -S 1048576 -t /test/certs 1 40C0882EE20C0000:error:80000002:system library:file_ctrl:No such file or directory:crypto/bio/bss_file.c:288:calling fopen(/test/certs/servercert.pem, r) 40C0882EE20C0000:error:10080002:BIO routines:file_ctrl:system lib:crypto/bio/bss_file.c:291: 40C0882EE20C0000:error:0A080002:SSL routines:SSL_CTX_use_certificate_file:system lib:ssl/ssl_rsa.c:329: /home/sashan/work.openssl/perftools/source/handshake.c:196: Failed to create SSL_CTX pair

My understanding is it is it expects CMAKE_PREFIX_PATH pointing to openssl sources but it got the path to custom installation. If I fix the CMAKE_PREFIX_PATH so it points to sources where custom openssl got installed from I get a different failure:

lifty$ make run [ 2%] Built target perf [ 3%] Built target writeread [ 3%] Run writeread -d -b 4096 /test/certs 4 Failed to create SSL_CTX pair *** Error 1 in . (CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/build.make:71 'CMakeFiles/run-writeread--d--b-4096--test-certs-4': /...) *** Error 2 in . (CMakeFiles/Makefile2:9133 'CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/all': make -s -f CMakeFiles/run-writeread...) *** Error 2 in . (CMakeFiles/Makefile2:1125 'CMakeFiles/run.dir/rule': make -s -f CMakeFiles/Makefile2 CMakeFiles/run.dir/all) *** Error 2 in /home/sashan/work.openssl/perftools/source/build (Makefile:306 'run': make -s -f CMakeFiles/Makefile2 run)

I understand the motivation to improve the CI experience. though I'm bit worried this makes the CMakeLists.txt bit too fancy to my taste. It was meant to be simple....

in ci runner it looks like the tool is doing the right thing. you may need to set DYLD_LIBRARY_PATH env. var explicitly in your shell when running cmake --build ... command, this may help.

In case of local runs with CMAKE_PREFIX_PATH being used, one can explicitly set run_certdir, cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install -Drun_certdir=/path/to/test/certs or so.

Sashan

I like the idea to improve CI tests. I just need more time to get familiar with cmake constructs which are used.

Sashan · 2025-10-17T17:11:04Z

source/CMakeLists.txt

+            endforeach()
+            set(cmds ${new_cmds})
+        endif()
+    endforeach()


Works fine on OpenBSD where I run cmake as follows:

cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install cmake --build build cmake --build build -t run

The build is setup such linker adds RUNPATH to elf header. The RUNPATH is set to /path/to/openssl.install I will take a look at macos.

In my case the failure happens later due to missing/wrong path to testcert dirs.

[ 42%] Built target run-evp-setpeer--k-ec256--t-4 [ 42%] Built target handshake [ 42%] Run handshake -P -S 1048576 -t /test/certs 1 40C0882EE20C0000:error:80000002:system library:file_ctrl:No such file or directory:crypto/bio/bss_file.c:288:calling fopen(/test/certs/servercert.pem, r) 40C0882EE20C0000:error:10080002:BIO routines:file_ctrl:system lib:crypto/bio/bss_file.c:291: 40C0882EE20C0000:error:0A080002:SSL routines:SSL_CTX_use_certificate_file:system lib:ssl/ssl_rsa.c:329: /home/sashan/work.openssl/perftools/source/handshake.c:196: Failed to create SSL_CTX pair

My understanding is it is it expects CMAKE_PREFIX_PATH pointing to openssl sources but it got the path to custom installation. If I fix the CMAKE_PREFIX_PATH so it points to sources where custom openssl got installed from I get a different failure:

lifty$ make run [ 2%] Built target perf [ 3%] Built target writeread [ 3%] Run writeread -d -b 4096 /test/certs 4 Failed to create SSL_CTX pair *** Error 1 in . (CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/build.make:71 'CMakeFiles/run-writeread--d--b-4096--test-certs-4': /...) *** Error 2 in . (CMakeFiles/Makefile2:9133 'CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/all': make -s -f CMakeFiles/run-writeread...) *** Error 2 in . (CMakeFiles/Makefile2:1125 'CMakeFiles/run.dir/rule': make -s -f CMakeFiles/Makefile2 CMakeFiles/run.dir/all) *** Error 2 in /home/sashan/work.openssl/perftools/source/build (Makefile:306 'run': make -s -f CMakeFiles/Makefile2 run)

I understand the motivation to improve the CI experience. though I'm bit worried this makes the CMakeLists.txt bit too fancy to my taste. It was meant to be simple....

Sashan

I think this useful. I let @esyr to polish the code (if needed) and re-submit for review once he will be happy with the code. thanks.

A helper function that prints version information, so far the version of the OpenSSL library a program is run with. Signed-off-by: Eugene Syromiatnikov <[email protected]>

When a test program is called with "-V" option, it prints version information and exits. Signed-off-by: Eugene Syromiatnikov <[email protected]>

Signed-off-by: Eugene Syromiatnikov <[email protected]>

Generate run-version-<test> target for each test from run_tests, make it a dependency of run-version target. and, if run_add_version_dep vesioble is set to ON (the default), set "run-version" as a dependency of the "run" target. Signed-off-by: Eugene Syromiatnikov <[email protected]>

…tage To increase readability and catch incorrect linking issues early in the run. Signed-off-by: Eugene Syromiatnikov <[email protected]>

Sashan · 2025-11-01T07:22:39Z

source/cmake/copy_reqs.cmake

@@ -0,0 +1,18 @@
+include(GetPrerequisites)


Perhaps I'm missing something... why do we need this? I would expect cmake's file copy as I prefer to keep things simple. thanks.

It's a CMake module[1] for getting a list of DLLs required by an executable, so they can be copied in the executable's directory. I tried cmake -E copy_if_different, but was keeping hitting issues on concurrent runs, so opted for copy_configuration instead, as it first copied a file into a temporary path next to destination, does the comparison, and then a (supposedly atomic) rename (and copy_if_different seems to do comparison and then copy).

[1] https://cmake.org/cmake/help/latest/module/GetPrerequisites.html

OK thanks for clarification. IMO doing copy $OPENSSL_SRC\*.dll $CMAKE_CURRENT_BINARY_DIR like copy all ddls from OPENSSL directory to perftools build dir. The reason I prefer doing simple/sufficient thing over taking the right/correct approach is we will have to pay attention to cmake version more closely as the current code here uses deprecated API. Starting cmake 3.16 one should be using file(GET_RUNTIME_DEPENDENCIES).

anyway. I think this should go in as-is. so we can get move on.

Deprecated since version 3.16: Use file(GET_RUNTIME_DEPENDENCIES) instead.

Ideally, we can even use $<TARGET_RUNTIME_DLLS>[1] (3.21+), but due to cmake_minimum_required(VERSION 3.10) we can't have nice things (while supporting pre-3.16/3.21 CMake, anyway).

[1] https://cmake.org/cmake/help/latest/manual/cmake-generator-expressions.7.html#genex:TARGET_RUNTIME_DLLS

Sashan

I think it's good enough to go in.

npajkovsky · 2025-11-03T07:17:23Z

.github/workflows/test.yml

+          }, {
+            openssl-branch: "master",
+            cmakeopts: '',
+            configopts: 'no-apps no-tests',


What's the reason that 3.{5,6} and master skips -D "run_newrawkey_algos=newrawkey;-a;x25519" -D "run_evp_fetch_pqs=evp_fetch;;" -D "run_handshake_pool_size=handshake;;;-o 4"?

Those are overrides for PQ-including defaults. I'm now thinking that those can just as well be moved to CMakeLists, at the expense of making it a bit more magical.

npajkovsky · 2025-11-03T07:18:10Z

source/cmake/copy_reqs.cmake

@@ -0,0 +1,18 @@
+include(GetPrerequisites)


Deprecated since version 3.16: Use file(GET_RUNTIME_DEPENDENCIES) instead.

npajkovsky · 2025-11-03T07:21:22Z

.github/workflows/test.yml

-          - "openssl-3.4"
-          - "openssl-3.5"
-          - "master"
+        release: [


Is there any reason why we need this? It prolongs the CI pipeline extensively.

There seems to be no other place where the tests are actually being run with various options. I plan to add universal support for the -T timeout option and run (the majority of) the test matrix with it, to alleviate the issue somewhat, but didn't want to pile on onto the existing patch set too much.

Introduce a run target, that executes all the test with different options. For now is used only for minimal sanity checking, that are test can be run and don't fail, but later can be extended to check the results for sanity as well, and the design has some provisions put in place to be useful for test data collection as well, for example. Reviewed-by: Neil Horman <[email protected]> (Merged from #53)

Sashan · 2025-11-06T12:47:36Z

change is in via 3c994d3, thanks.

esyr force-pushed the esyr/cmake-run branch 8 times, most recently from 67580ed to e693b26 Compare October 16, 2025 19:13

esyr changed the title ~~Add cmake run targets and use it in PR CI~~ Add cmake "run" target and use it in PR CI Oct 16, 2025

esyr requested review from Sashan, jogme, nhorman and quarckster October 16, 2025 19:41

jogme approved these changes Oct 17, 2025

View reviewed changes

npajkovsky reviewed Oct 17, 2025

View reviewed changes

Sashan reviewed Oct 17, 2025

View reviewed changes

Sashan force-pushed the esyr/cmake-run branch 2 times, most recently from a2d538e to bbddd22 Compare October 19, 2025 23:43

Sashan added this to Project Board Oct 20, 2025

github-project-automation bot moved this to New in Project Board Oct 20, 2025

Sashan moved this from New to Awaits review in Project Board Oct 20, 2025

Sashan removed this from Project Board Oct 20, 2025

Sashan added this to Development Board Oct 20, 2025

Sashan moved this to Waiting Review in Development Board Oct 20, 2025

vavroch2010 assigned esyr Oct 20, 2025

Sashan requested changes Oct 21, 2025

View reviewed changes

vavroch2010 moved this from Waiting Review to In Progress in Development Board Oct 21, 2025

esyr force-pushed the esyr/cmake-run branch 2 times, most recently from e512c45 to f1b9dc1 Compare October 30, 2025 16:22

esyr added 3 commits November 1, 2025 03:26

perflib: add perflib_print_version()

618a6c6

A helper function that prints version information, so far the version of the OpenSSL library a program is run with. Signed-off-by: Eugene Syromiatnikov <[email protected]>

source: wire up the -V option to all tests

7071827

When a test program is called with "-V" option, it prints version information and exits. Signed-off-by: Eugene Syromiatnikov <[email protected]>

evp_hash: exit successfully when called with -h option

f698212

Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr force-pushed the esyr/cmake-run branch from 07aad7f to 36dfea3 Compare November 1, 2025 02:28

esyr added 2 commits November 1, 2025 03:32

.github/workflows/test.yml: print version information in a separate s…

34e5d4e

…tage To increase readability and catch incorrect linking issues early in the run. Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr force-pushed the esyr/cmake-run branch from 36dfea3 to 34e5d4e Compare November 1, 2025 02:33

esyr requested review from Sashan, jogme and npajkovsky November 1, 2025 03:04

Sashan reviewed Nov 1, 2025

View reviewed changes

Sashan approved these changes Nov 2, 2025

View reviewed changes

github-project-automation bot moved this from In Progress to Waiting Merge in Development Board Nov 2, 2025

npajkovsky reviewed Nov 3, 2025

View reviewed changes

esyr mentioned this pull request Nov 5, 2025

Refactor perftests argument parsing a bit openssl/project#1712

Open

nhorman added approval: ready to merge The 24 hour grace period has passed, ready to merge approval: done This pull request has the required number of approvals and removed approval: ready to merge The 24 hour grace period has passed, ready to merge labels Nov 5, 2025

nhorman moved this from Waiting Merge to Waiting Review in Development Board Nov 5, 2025

nhorman added approval: review pending This pull request needs review by a committer and removed approval: done This pull request has the required number of approvals labels Nov 5, 2025

nhorman approved these changes Nov 5, 2025

View reviewed changes

github-project-automation bot moved this from Waiting Review to Waiting Merge in Development Board Nov 5, 2025

nhorman added approval: done This pull request has the required number of approvals and removed approval: review pending This pull request needs review by a committer labels Nov 5, 2025

npajkovsky approved these changes Nov 6, 2025

View reviewed changes

Sashan closed this Nov 6, 2025

github-project-automation bot moved this from Waiting Merge to Done in Development Board Nov 6, 2025

Add cmake "run" target and use it in PR CI #53

Add cmake "run" target and use it in PR CI #53

Uh oh!

Conversation

esyr commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jogme left a comment

Choose a reason for hiding this comment

Uh oh!

quarckster commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sashan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sashan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esyr Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sashan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esyr Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sashan commented Nov 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

esyr commented Oct 16, 2025 •

edited

Loading

esyr Nov 3, 2025 •

edited

Loading

esyr Nov 3, 2025 •

edited

Loading