Add cmake "run" target and use it in PR CI #53

esyr · 2025-10-16T16:41:16Z

Introduce a run target, that executes all the test with different options. For now is used only for minimal sanity checking, that are test can be run and don't fail, but later can be extended to check the results for sanity as well, and the design has some provisions put in place to be useful for test data collection as well, for example.

jogme

lgtm, thank you!

quarckster · 2025-10-17T08:42:17Z

Interesting that the only Windows on master didn't pass.

npajkovsky · 2025-10-17T08:43:30Z

source/CMakeLists.txt

+            endforeach()
+            set(cmds ${new_cmds})
+        endif()
+    endforeach()


I have tried it but make run does not respect or set (DY)LD_LIBRARY_PATH which should be fixed.

I would probably write custom target manually for each binary. The cmd options are much too different for a generic solution.

$ echo $DYLD_LIBRARY_PATH /Users/npajkovsky/openssl/openssl $ ll /Users/npajkovsky/openssl/openssl | grep ssl.4 -rwxr-xr-x 1 npajkovsky staff 960K Oct 17 10:21 libssl.4.dylib $ make run [ 2%] Built target perf [ 3%] Built target x509storeissuer [ 3%] Run x509storeissuer -t /Users/npajkovsky/openssl/openssl/test/certs 4 dyld[58574]: Library not loaded: /usr/local/lib/libssl.4.dylib Referenced from: <7C191CC0-3566-3B84-9EB4-9AB01FD44703> /Users/npajkovsky/openssl/perftools/source/x509storeissuer Reason: tried: '/usr/local/lib/libssl.4.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/lib/libssl.4.dylib' (no such file), '/usr/local/lib/libssl.4.dylib' (no such file) make[3]: *** [CMakeFiles/run-x509storeissuer--t--U_sers-npajkovsky-openssl-openssl-test-certs-4] Abort trap: 6 make[2]: *** [CMakeFiles/run-x509storeissuer--t--U_sers-npajkovsky-openssl-openssl-test-certs-4.dir/all] Error 2 make[1]: *** [CMakeFiles/run.dir/rule] Error 2 make: *** [run] Error 2

Have you exported the variable? But that's a valid point, that cmake doesn't do any provisions to supply the provided OPENSSL_ROOT_DIR to the executables being run, the relevant overrides should be added to the CI recipes.

Works fine on OpenBSD where I run cmake as follows:

cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install cmake --build build cmake --build build -t run

The build is setup such linker adds RUNPATH to elf header. The RUNPATH is set to /path/to/openssl.install I will take a look at macos.

In my case the failure happens later due to missing/wrong path to testcert dirs.

[ 42%] Built target run-evp-setpeer--k-ec256--t-4 [ 42%] Built target handshake [ 42%] Run handshake -P -S 1048576 -t /test/certs 1 40C0882EE20C0000:error:80000002:system library:file_ctrl:No such file or directory:crypto/bio/bss_file.c:288:calling fopen(/test/certs/servercert.pem, r) 40C0882EE20C0000:error:10080002:BIO routines:file_ctrl:system lib:crypto/bio/bss_file.c:291: 40C0882EE20C0000:error:0A080002:SSL routines:SSL_CTX_use_certificate_file:system lib:ssl/ssl_rsa.c:329: /home/sashan/work.openssl/perftools/source/handshake.c:196: Failed to create SSL_CTX pair

My understanding is it is it expects CMAKE_PREFIX_PATH pointing to openssl sources but it got the path to custom installation. If I fix the CMAKE_PREFIX_PATH so it points to sources where custom openssl got installed from I get a different failure:

lifty$ make run [ 2%] Built target perf [ 3%] Built target writeread [ 3%] Run writeread -d -b 4096 /test/certs 4 Failed to create SSL_CTX pair *** Error 1 in . (CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/build.make:71 'CMakeFiles/run-writeread--d--b-4096--test-certs-4': /...) *** Error 2 in . (CMakeFiles/Makefile2:9133 'CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/all': make -s -f CMakeFiles/run-writeread...) *** Error 2 in . (CMakeFiles/Makefile2:1125 'CMakeFiles/run.dir/rule': make -s -f CMakeFiles/Makefile2 CMakeFiles/run.dir/all) *** Error 2 in /home/sashan/work.openssl/perftools/source/build (Makefile:306 'run': make -s -f CMakeFiles/Makefile2 run)

I understand the motivation to improve the CI experience. though I'm bit worried this makes the CMakeLists.txt bit too fancy to my taste. It was meant to be simple....

in ci runner it looks like the tool is doing the right thing. you may need to set DYLD_LIBRARY_PATH env. var explicitly in your shell when running cmake --build ... command, this may help.

In case of local runs with CMAKE_PREFIX_PATH being used, one can explicitly set run_certdir, cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install -Drun_certdir=/path/to/test/certs or so.

Sashan

I like the idea to improve CI tests. I just need more time to get familiar with cmake constructs which are used.

Sashan · 2025-10-17T17:11:04Z

source/CMakeLists.txt

+            endforeach()
+            set(cmds ${new_cmds})
+        endif()
+    endforeach()


Works fine on OpenBSD where I run cmake as follows:

cmake -S . -B ./build -DOPENSSL_CONFIG_MODE=1 -DCMAKE_PREFIX_PATH=/path/to/openssl.install cmake --build build cmake --build build -t run

The build is setup such linker adds RUNPATH to elf header. The RUNPATH is set to /path/to/openssl.install I will take a look at macos.

In my case the failure happens later due to missing/wrong path to testcert dirs.

[ 42%] Built target run-evp-setpeer--k-ec256--t-4 [ 42%] Built target handshake [ 42%] Run handshake -P -S 1048576 -t /test/certs 1 40C0882EE20C0000:error:80000002:system library:file_ctrl:No such file or directory:crypto/bio/bss_file.c:288:calling fopen(/test/certs/servercert.pem, r) 40C0882EE20C0000:error:10080002:BIO routines:file_ctrl:system lib:crypto/bio/bss_file.c:291: 40C0882EE20C0000:error:0A080002:SSL routines:SSL_CTX_use_certificate_file:system lib:ssl/ssl_rsa.c:329: /home/sashan/work.openssl/perftools/source/handshake.c:196: Failed to create SSL_CTX pair

My understanding is it is it expects CMAKE_PREFIX_PATH pointing to openssl sources but it got the path to custom installation. If I fix the CMAKE_PREFIX_PATH so it points to sources where custom openssl got installed from I get a different failure:

lifty$ make run [ 2%] Built target perf [ 3%] Built target writeread [ 3%] Run writeread -d -b 4096 /test/certs 4 Failed to create SSL_CTX pair *** Error 1 in . (CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/build.make:71 'CMakeFiles/run-writeread--d--b-4096--test-certs-4': /...) *** Error 2 in . (CMakeFiles/Makefile2:9133 'CMakeFiles/run-writeread--d--b-4096--test-certs-4.dir/all': make -s -f CMakeFiles/run-writeread...) *** Error 2 in . (CMakeFiles/Makefile2:1125 'CMakeFiles/run.dir/rule': make -s -f CMakeFiles/Makefile2 CMakeFiles/run.dir/all) *** Error 2 in /home/sashan/work.openssl/perftools/source/build (Makefile:306 'run': make -s -f CMakeFiles/Makefile2 run)

I understand the motivation to improve the CI experience. though I'm bit worried this makes the CMakeLists.txt bit too fancy to my taste. It was meant to be simple....

Sashan

I think this useful. I let @esyr to polish the code (if needed) and re-submit for review once he will be happy with the code. thanks.

Add "run" target that runs all the tests with various options. Co-Authored-by: Alexandr Nedvedicky <[email protected]> Signed-off-by: Eugene Syromiatnikov <[email protected]>

Signed-off-by: Eugene Syromiatnikov <[email protected]>

These take time (especially on Windows) and we don't use them here anyway. Signed-off-by: Eugene Syromiatnikov <[email protected]>

A helper function that prints version information, so far the version of the OpenSSL library a program is run with. Signed-off-by: Eugene Syromiatnikov <[email protected]>

When a test program is called with "-V" option, it prints version information and exits. Signed-off-by: Eugene Syromiatnikov <[email protected]>

Signed-off-by: Eugene Syromiatnikov <[email protected]>

Generate run-version-<test> target for each test from run_tests, make it a dependency of run-version target. and, if run_add_version_dep vesioble is set to ON (the default), set "run-version" as a dependency of the "run" target. Signed-off-by: Eugene Syromiatnikov <[email protected]>

…tage To increase readability and catch incorrect linking issues early in the run. Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr force-pushed the esyr/cmake-run branch 8 times, most recently from 67580ed to e693b26 Compare October 16, 2025 19:13

esyr changed the title ~~Add cmake run targets and use it in PR CI~~ Add cmake "run" target and use it in PR CI Oct 16, 2025

esyr requested review from Sashan, jogme, nhorman and quarckster October 16, 2025 19:41

jogme approved these changes Oct 17, 2025

View reviewed changes

npajkovsky reviewed Oct 17, 2025

View reviewed changes

Sashan reviewed Oct 17, 2025

View reviewed changes

Sashan force-pushed the esyr/cmake-run branch 2 times, most recently from a2d538e to bbddd22 Compare October 19, 2025 23:43

Sashan added this to Project Board Oct 20, 2025

github-project-automation bot moved this to New in Project Board Oct 20, 2025

Sashan moved this from New to Awaits review in Project Board Oct 20, 2025

Sashan removed this from Project Board Oct 20, 2025

Sashan added this to Development Board Oct 20, 2025

Sashan moved this to Waiting Review in Development Board Oct 20, 2025

vavroch2010 assigned esyr Oct 20, 2025

Sashan requested changes Oct 21, 2025

View reviewed changes

vavroch2010 moved this from Waiting Review to In Progress in Development Board Oct 21, 2025

Sashan and others added 2 commits October 30, 2025 16:32

evp_fetch: actually iterate on the fetch_entries array

25b4fb1

source/CMakeLists.txt: add run target

49fa016

Add "run" target that runs all the tests with various options. Co-Authored-by: Alexandr Nedvedicky <[email protected]> Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr added 4 commits October 30, 2025 16:33

.github/workflows/test.yml: execture cmake run target in CI runs

a2da967

Signed-off-by: Eugene Syromiatnikov <[email protected]>

.github/workflows/test.yml: add openssl-3.6 branch to CI test matrix

a6e8f03

Signed-off-by: Eugene Syromiatnikov <[email protected]>

.github/workflows/test.yml: do not build OpenSSL apps and tests

41f5631

These take time (especially on Windows) and we don't use them here anyway. Signed-off-by: Eugene Syromiatnikov <[email protected]>

perflib: add perflib_print_version()

523c727

A helper function that prints version information, so far the version of the OpenSSL library a program is run with. Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr force-pushed the esyr/cmake-run branch from 1f8c4af to 5a1a911 Compare October 30, 2025 15:35

esyr added 4 commits October 30, 2025 17:22

source: wire up the -V option to all tests

95dad55

When a test program is called with "-V" option, it prints version information and exits. Signed-off-by: Eugene Syromiatnikov <[email protected]>

evp_hash: exit successfully when called with -h option

6746201

Signed-off-by: Eugene Syromiatnikov <[email protected]>

.github/workflows/test.yml: print version information in a separate s…

f1b9dc1

…tage To increase readability and catch incorrect linking issues early in the run. Signed-off-by: Eugene Syromiatnikov <[email protected]>

esyr force-pushed the esyr/cmake-run branch from e512c45 to f1b9dc1 Compare October 30, 2025 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add cmake "run" target and use it in PR CI #53

Add cmake "run" target and use it in PR CI #53

esyr commented Oct 16, 2025 •

edited

Loading

Uh oh!

jogme left a comment

Uh oh!

quarckster commented Oct 17, 2025

Uh oh!

npajkovsky Oct 17, 2025

Uh oh!

esyr Oct 17, 2025

Uh oh!

Sashan Oct 17, 2025

Uh oh!

Sashan Oct 20, 2025

Uh oh!

esyr Oct 24, 2025

Uh oh!

Sashan left a comment

Uh oh!

Sashan Oct 17, 2025

Uh oh!

Sashan left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add cmake "run" target and use it in PR CI #53

Are you sure you want to change the base?

Add cmake "run" target and use it in PR CI #53

Conversation

esyr commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jogme left a comment

Choose a reason for hiding this comment

Uh oh!

quarckster commented Oct 17, 2025

Uh oh!

npajkovsky Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

esyr Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Sashan Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Sashan Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

esyr Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Sashan left a comment

Choose a reason for hiding this comment

Uh oh!

Sashan Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Sashan left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

esyr commented Oct 16, 2025 •

edited

Loading