Skip to content

Commit 094cddc

Browse files
alandefreitasalandefreitas
committed
Alan's post on B2-Style Test Granularity
1 parent e0b5a2b commit 094cddc

File tree

1 file changed

+254
-0
lines changed

1 file changed

+254
-0
lines changed

‎_posts/2025-07-10-Alan.md‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
---
2+
layout: post
3+
nav-class: dark
4+
categories: [ alan ]
5+
title: Bringing B2-Style Test Granularity to CMake
6+
author-id: alan
7+
author-name: Alan de Freitas
8+
---
9+
10+
# Introduction
11+
12+
Boost libraries typically maintain granular unit tests using Boost.Build (B2). B2 provides a `run` rule that makes it easy to define many independent test targets from a single source file or executable. Each test case can be listed, invoked, and reported separately, which improves developer workflow, test clarity, and CI diagnostics.
13+
14+
However, Boost’s CMake integration has lacked this granularity. When Boost libraries are built with CMake, the typical approach is to define a single test executable and add all test suites as a single test in CTest with `add_test()`. As a result, when running tests with CTest, developers lose the ability to see individual test failures in isolation, run only subsets of tests, or leverage parallel execution at the test level.
15+
16+
The goal of this work is to bridge that gap. We want to replicate the B2 “one executable, many independent tests” idiom in CMake. Specifically, we want to use modern CMake techniques to split a single unit test executable into multiple independent CTest targets, while preserving the flexibility and simplicity of the Boost testing style.
17+
18+
# Problem Analysis
19+
20+
To understand why splitting tests into independent CTest targets is non-trivial, it helps to look at **how CMake's build and test model is structured**.
21+
22+
When building and testing libraries with CMake, the developer usually has a workflow of four key phases:
23+
24+
- **Configuration step**: This is where CMakeLists.txt files are processed and commands like `add_executable` and `add_test()` are called. Test targets must be defined here, so CTest knows about them.
25+
- **Build step**: This is when the underlying build system (the CMake "generator": e.g., Ninja or Make) compiles sources and produces executables, including unit test binaries.
26+
- **Test step**: This is when `ctest` runs the defined tests, using the executables built in the previous step.
27+
- **Installation step**: This is where the built libraries and executables are installed to their final locations. This step is not directly relevant to the problem at hand, but it’s part of the overall CMake workflow.
28+
29+
At the configuration step, we would like to have something like
30+
31+
```cmake
32+
add_test(NAME test_a COMMAND my_unit_test_executable a)
33+
add_test(NAME test_b COMMAND my_unit_test_executable b)
34+
add_test(NAME test_c COMMAND my_unit_test_executable c)
35+
```
36+
37+
instead of
38+
39+
```cmake
40+
add_test(NAME my_unit_test_executable COMMAND my_unit_test_executable)
41+
```
42+
43+
The fundamental obstacle is that, in the general case, **you cannot know what tests exist inside a unit test executable until you can *run* it**. Many modern test frameworks (like Boost.Test, Catch2, GoogleTest) support listing their available tests by running the executable with a special argument (e.g., `--list-tests`). But this only works *after* the executable is built.
44+
45+
In other words:
46+
47+
> You need the executable to discover the tests, but CMake requires you to declare the tests in the configuration phase before building the executable in the build step.
48+
49+
This dependency cycle is the core problem that makes it difficult to reproduce B2's `run` rule semantics in CMake. Without special handling, you’re forced to treat the entire unit test binary as a single test, losing the ability to register its internal test cases as independent CTest targets.
50+
51+
The solution to this problem involves the [`TEST_INCLUDE_FILES`](https://cmake.org/cmake/help/latest/prop_dir/TEST_INCLUDE_FILES.html) directory property, which allows you to specify additional files that CTest should consider when running tests. By leveraging this property, we can dynamically generate a CMake script that defines individual `add_test()` calls for each test case found in the unit test executable.
52+
53+
So we can use
54+
55+
```cmake
56+
set_property(DIRECTORY
57+
APPEND PROPERTY TEST_INCLUDE_FILES "${TEST_SUITE_CTEST_INCLUDE_FILE}"
58+
)
59+
```
60+
61+
to include a generated CMake script that contains the individual test definitions. This allows us to run the executable post-build, extract the test names, and then register them with CTest in a way that mimics the B2 experience. Other modern test frameworks have explored this feature to provide an automated test discovery mechanism for their libraries in CMake. For example, Catch2’s `catch_discover_tests()` and GoogleTest’s `gtest_discover_tests()` run the built test executable with a listing flag (like `--list-tests`) to extract individual test cases and generate separate `add_test()` entries for each.
62+
63+
# Design Overview
64+
65+
Since CMake requires test registration in the configuration step, but we can only discover test cases *after* building the executable, we introduce an approach to bridge that gap. The high-level plan is:
66+
67+
- **Build the executable**: Compile the unit test executable as usual. The target is defined in the configuration step and built in the build step.
68+
- **Post-build step**: After building, run the executable with `--list-tests` (or equivalent) to enumerate all available test cases. This is achieved with a custom command that runs after the build completes.
69+
- **Generate a CMake script**: This post-build step writes a `.cmake` file containing one `add_test()` call for each discovered test case.
70+
- **Conditional inclusion**: The main CMake configuration includes this generated script *only if it exists*, so the tests appear in CTest after they’re generated. The new script is included using the `TEST_INCLUDE_FILES` property, which allows CTest to pick it up automatically.
71+
72+
This approach effectively moves test discovery to the build phase while still registering the resulting tests with CTest in the configuration phase for subsequent runs.
73+
74+
This process is transparent to the user. In Boost.URL, where we implemented the functionality, the test registration process went from:
75+
76+
```cmake
77+
add_test(NAME boost_url_unit_tests COMMAND boost_url_unit_tests)
78+
```
79+
80+
to
81+
82+
```cmake
83+
boost_url_test_suite_discover_tests(boost_url_unit_tests)
84+
```
85+
86+
# Implementation Details
87+
88+
This section describes the approach bottom-up, showing the overall mechanism of discovering and registering independent test targets in CMake.
89+
90+
## The Test Listing Extractor Script
91+
92+
The first piece is a small CMake script that **runs the compiled test executable** with `--list-tests` (or an equivalent flag your test framework supports). It captures the output, which is expected to be a plain list of test case names.
93+
94+
For example, suppose your unit test executable outputs:
95+
96+
```cmake
97+
UnitA.TestAlpha
98+
UnitA.TestBeta
99+
UnitB.TestGamma
100+
```
101+
102+
The script saves these names so they can be transformed into separate CTest targets.
103+
104+
Example command in CMake:
105+
106+
```cmake
107+
execute_process(
108+
COMMAND "${TEST_SUITE_TEST_EXECUTABLE}" ${TEST_SUITE_TEST_SPEC} --list-tests
109+
OUTPUT_VARIABLE TEST_SUITE_LIST_TESTS_OUTPUT
110+
ERROR_VARIABLE TEST_SUITE_LIST_TESTS_OUTPUT
111+
RESULT_VARIABLE TEST_SUITE_RESULT
112+
WORKING_DIRECTORY "${TEST_SUITE_TEST_WORKING_DIR}"
113+
)
114+
```
115+
116+
## Generator of CMake Test Definitions
117+
118+
Once the list of tests is available, the script generates a new `.cmake` file containing one `add_test()` call per discovered test. This file effectively defines the independent CTest targets.
119+
120+
Example generated `tests.cmake` content:
121+
122+
```cmake
123+
add_test(NAME UnitA.TestAlpha COMMAND my_test_executable UnitA.TestAlpha)
124+
add_test(NAME UnitA.TestBeta COMMAND my_test_executable UnitA.TestBeta)
125+
add_test(NAME UnitB.TestGamma COMMAND my_test_executable UnitB.TestGamma)
126+
```
127+
128+
This approach ensures each test is addressable, selectable, and independently reported by CTest.
129+
130+
## Post-Build Step Integration
131+
132+
CMake can't know these test names at configuration time, so we hook the test listing step to the build phase using a `POST_BUILD` custom command. After the test executable is built, this command runs the extractor and generates the script file defining the tests.
133+
134+
Example:
135+
136+
```cmake
137+
add_custom_command(
138+
# The executable target with the unit tests
139+
TARGET ${TARGET}
140+
POST_BUILD
141+
BYPRODUCTS "${TEST_SUITE_CTEST_TESTS_FILE}"
142+
# Run the CMake script to discover tests after the build step
143+
COMMAND "${CMAKE_COMMAND}"
144+
# Arguments to the script
145+
-D "TEST_TARGET=${TARGET}"
146+
-D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
147+
-D "TEST_WORKING_DIR=${TEST_SUITE_WORKING_DIRECTORY}"
148+
# ...
149+
# The output file where the test definitions will be written
150+
-D "CTEST_FILE=${TEST_SUITE_CTEST_TESTS_FILE}"
151+
# The script that generates the test definitions
152+
-P "${TEST_SUITE_DISCOVER_AND_WRITE_TESTS_SCRIPT}"
153+
VERBATIM
154+
)
155+
```
156+
157+
This ensures the test listing happens automatically as part of the build.
158+
159+
## Including Generated Tests
160+
161+
The main CMake configuration includes the generated `.cmake` file, but only if it exists. This avoids errors on a test pass before the executable is built. This could happen because the user is calling `ctest` before the build step completes, because the test executable was not built, or because the cache was invalidated.
162+
163+
So the discovery function uses the example pattern:
164+
165+
```cmake
166+
if(EXISTS "${CMAKE_BINARY_DIR}/generated/tests.cmake")
167+
include("${CMAKE_BINARY_DIR}/generated/tests.cmake")
168+
endif()
169+
```
170+
171+
And this is the file that the test step will ultimately include in the CTest run, allowing CTest to see all the individual test targets.
172+
173+
## CMake Function for Reuse
174+
175+
To make this easy for other libraries, the pattern can be wrapped in a CMake function. This function:
176+
177+
* Defines the `POST_BUILD` rule for the given target.
178+
* Encapsulates the details of running the extractor script.
179+
* Ensures consistent output locations for the generated test definitions.
180+
181+
Example usage:
182+
183+
```cmake
184+
boost_url_test_suite_discover_tests(boost_url_unit_tests)
185+
```
186+
187+
This approach lets library maintainers adopt the system with minimal changes to their existing CMake setup, while maintaining Boost’s fine-grained, many-target test philosophy.
188+
189+
When we look at CI results for Boost.URL, this is the only thing we used to have:
190+
191+
```
192+
/__w/_tool/cmake/3.20.0/x64/bin/ctest --test-dir /__w/url/boost-root/build_cmake --parallel 4 --no-tests=error --progress --output-on-failure
193+
Internal ctest changing into directory: /__w/url/boost-root/build_cmake
194+
Test project /__w/url/boost-root/build_cmake
195+
Start 1: boost_url_unit_tests
196+
Start 2: boost_url_extra
197+
Start 3: boost_url_limits
198+
1/3 Test #2: boost_url_extra .................. Passed 0.00 sec
199+
2/3 Test #3: boost_url_limits ................. Passed 0.00 sec
200+
3/3 Test #1: boost_url_unit_tests ............. Passed 0.02 sec
201+
202+
100% tests passed, 0 tests failed out of 3
203+
204+
Total Test time (real) = 0.02 sec
205+
```
206+
207+
And now we see one unit test per test case:
208+
209+
```
210+
/__w/_tool/cmake/3.20.0/x64/bin/ctest --test-dir /__w/url/boost-root/build_cmake --parallel 4 --no-tests=error --progress --output-on-failure
211+
Internal ctest changing into directory: /__w/url/boost-root/build_cmake
212+
Test project /__w/url/boost-root/build_cmake
213+
Start 1: boost.url.absolute_uri_rule
214+
Start 2: boost.url.authority_rule
215+
Start 3: boost.url.authority_view
216+
Start 4: boost.url.compat.ada
217+
1/76 Test #1: boost.url.absolute_uri_rule .......... Passed 0.01 sec
218+
Start 5: boost.url.decode_view
219+
2/76 Test #2: boost.url.authority_rule ............. Passed 0.01 sec
220+
Start 6: boost.url.doc.3_urls
221+
3/76 Test #3: boost.url.authority_view ............. Passed 0.01 sec
222+
Start 7: boost.url.doc.grammar
223+
4/76 Test #5: boost.url.decode_view ................ Passed 0.01 sec
224+
Start 8: boost.url.encode
225+
5/76 Test #4: boost.url.compat.ada ................. Passed 0.01 sec
226+
Start 9: boost.url.error
227+
6/76 Test #6: boost.url.doc.3_urls ................. Passed 0.01 sec
228+
Start 10: boost.url.format
229+
7/76 Test #7: boost.url.doc.grammar ................ Passed 0.01 sec
230+
Start 11: boost.url.gen_delim_chars
231+
8/76 Test #8: boost.url.encode ..................... Passed 0.01 sec
232+
Start 12: boost.url.grammar.alnum_chars
233+
...
234+
```
235+
236+
meaning that each test is now executed and reported individually and in parallel, allowing developers to see which specific tests passed or failed, and enabling more granular control over test execution.
237+
238+
# Conclusion
239+
240+
This approach brings fine-grained tests into the modern CMake Boost workflow. By splitting a single test executable into multiple independent CTest targets, maintainers gain:
241+
242+
- **More granular failure reporting**: CI logs show exactly which test case failed.
243+
- **Better developer experience**: Developers can run or re-run individual tests easily.
244+
- **Improved parallel execution**: Faster test runs in CI and locally.
245+
- **Better IDE integration**: IDEs can show individual test cases.
246+
247+
For other Boost libraries considering adopting this pattern, the only requirement is that their test executables support a `--list-tests` (or equivalent) command that outputs the available test cases. Once that's available, the necessary CMake changes to define an equivalent function are minimal:
248+
249+
- Add a `POST_BUILD` step that runs the listing command and generates the `.cmake` file.
250+
- Conditionally include that generated file in the main CMakeLists.
251+
252+
If the output of `--list-tests` is one test suite per line, the existing script can be used as-is. This small investment pays off with a much more maintainable and CI-friendly testing setup. I encourage other maintainers and contributors to try this technique, refine it, and share feedback.
253+
254+
The complete script and CMake snippets are available in the Boost.URL repository at commit [#a1a5d18](https://github.com/boostorg/url/commit/a1a5d18e9356036e446f98fc774eb1a1f5e242af).

0 commit comments

Comments
 (0)