Skip to content

Commit ccc9b0c

Browse files
cuinixamcuinixam
committed
feat: add ctest test discovery in vs code
1 parent 516fd22 commit ccc9b0c

File tree

3 files changed

+254
-0
lines changed

3 files changed

+254
-0
lines changed

‎docs/blogs/2025/images/all_tests.png‎

44 KB
Loading

‎docs/blogs/2025/images/not_build_tests.png‎

42.3 KB
Loading

‎docs/blogs/2025/vs_code_ctest.md‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
---
2+
tags: cmake, ctest, gtest, vs code, testing
3+
category: learning
4+
date: 2025-02-05
5+
title: Making Unit Tests Visible in Visual Studio Code with CTest and GoogleTest
6+
---
7+
8+
# Making Unit Tests Visible in Visual Studio Code with CTest and GoogleTest
9+
10+
## Introduction
11+
12+
Many C/C++ projects rely on CMake, GoogleTest (GTest), and GoogleMock (GMock) for unit testing.
13+
Sometimes developers want to see and run/debug these tests directly in their IDE.
14+
15+
In this blog post, I will show how to make the unit tests visible in Visual Studio Code (VS Code) using CTest and GoogleTest.
16+
I will start by explaining the standard way to build and run the tests with CMake.
17+
Then we will see how CTest can help us discover the tests and how to display them in the VS Code interface.
18+
19+
## Building and Running Tests with CMake and GoogleTest
20+
21+
In a typical setup one defines executables for each "component" with its own sources and unit tests files.
22+
There are CMake custom targets for every component to link the component sources and the test sources and execute the tests.
23+
24+
An instructive example of a CMakeLists.txt file is shown below:
25+
26+
```{code-block} cmake
27+
:linenos:
28+
29+
cmake_minimum_required(VERSION 3.20)
30+
project(MyProject)
31+
32+
set(CMAKE_CXX_STANDARD 14)
33+
set(CMAKE_CXX_STANDARD_REQUIRED ON)
34+
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
35+
36+
# Add GoogleTest components
37+
add_subdirectory(${CMAKE_SOURCE_DIR}/build/external/gtest ${CMAKE_BUILD_DIR}/.gtest)
38+
39+
# Configure directories to search for headers
40+
include_directories(
41+
${CMAKE_SOURCE_DIR}
42+
${CMAKE_SOURCE_DIR}/build/external/gtest/googletest/include
43+
${CMAKE_SOURCE_DIR}/build/external/gtest/googlemock/include
44+
)
45+
46+
# Create the test executable
47+
add_executable(greeter greeter.c greeter_test.cc)
48+
49+
target_link_libraries(greeter GTest::gtest_main GTest::gmock_main pthread)
50+
target_compile_options(greeter PRIVATE -ggdb --coverage)
51+
target_link_options(greeter PRIVATE --coverage)
52+
53+
# Run the test executable, generate JUnit report and return success independent of the test result
54+
add_custom_target(greeter_test ALL
55+
BYPRODUCTS ${CMAKE_BUILD_DIR}/greeter_junit.xml
56+
DEPENDS greeter
57+
COMMAND greeter --gtest_output="xml:greeter_junit.xml" || ${CMAKE_COMMAND} -E true
58+
)
59+
60+
```
61+
62+
What this basically does is:
63+
64+
- create a test executable `greeter` that links the sources `greeter.c` and `greeter_test.cc` with the GoogleTest libraries
65+
- create a custom target `greeter_test` that runs the test executable and generates a JUnit report
66+
67+
To build and run the tests, one would typically execute the following commands:
68+
69+
```bash
70+
cmake -B build -S .
71+
cmake --build build
72+
```
73+
74+
This will build all targets and generate the test executable `greeter` and the JUnit report `greeter_junit.xml`.
75+
76+
```{code-block} xml
77+
:linenos:
78+
:caption: greeter_junit.xml
79+
80+
<?xml version="1.0" encoding="UTF-8"?>
81+
<testsuites tests="2" failures="1" disabled="0" errors="0" time="0." name="AllTests">
82+
<testsuite name="GreeterTest" tests="2" failures="1" disabled="0" skipped="0" errors="0" time="0." >
83+
<testcase name="Greeting1" file="greeter_test.cc" line="14" status="run" result="completed" time="0." classname="GreeterTest" />
84+
</testsuite>
85+
</testsuites>
86+
87+
```
88+
89+
## Discovering Tests
90+
91+
Let us first shortly discuss on how one would discover the tests in the project.
92+
93+
Having the tests source files, one could try to parse them and extract the test cases.
94+
However, this is not a trivial task and it is error-prone.
95+
Imagine that a test is only enabled if a certain preprocessor definition is set.
96+
One should properly parse the (preprocessed) test source files to determine the list of relevant tests.
97+
I am not even touching on how creative some developers might be on creating their on macros to define tests.
98+
99+
Another approach is to "ask" the test executable to list the tests.
100+
This has the disadvantage that the test executable must be built and run.
101+
On the good side, the test executable knows exactly which tests are available and can provide additional information about them.
102+
103+
Indeed, the GTTest framework provides a way to list the tests.
104+
Running the test executable with the `--gtest_list_tests` flag will list all the tests and their test cases.
105+
106+
```
107+
greeter.exe --gtest_list_tests
108+
```
109+
110+
```{note}
111+
Calling the test executable with the `--help` option will print all the available options.
112+
```
113+
114+
We will go with the second approach and determine the tests by running the test executable.
115+
CMake has a built-in tool called CTest that can help us with this.
116+
117+
### Defining and running tests with CTest
118+
119+
CTest is a testing tool that is part of the CMake suite.
120+
CMake facilitates testing through special commands and the CTest executable.
121+
122+
The integration is quite simple. One needs to "enable testing" and define tests using the `add_test` command:
123+
124+
```{code-block} cmake
125+
include(CTest) # automatically calls enable_testing()
126+
add_test(NAME greeter_test COMMAND greeter)
127+
```
128+
129+
The `add_test` command has a simple syntax:
130+
131+
- `NAME` - the name of the test, can be any string
132+
- `COMMAND` - the command to run the test, in our case the test executable `greeter`. This can be any command that is run in the shell and returns a status code (non-zero for failure).
133+
134+
To run the tests, one would execute the following command:
135+
136+
```
137+
cd build
138+
ctest
139+
```
140+
141+
CTest has many options to control the test execution, like running only a specific test, running tests in parallel, or generating a JUnit report.
142+
For more information check the help or the documentation.
143+
144+
#### Details on how CMake/CTest work
145+
146+
When testing is enabled, CMake generates a `CTestTestfile.cmake` file in the build directory.
147+
This file will include all the tests defined with the `add_test` command.
148+
When running `ctest`, CTest will read this file and execute the tests.
149+
150+
```{code-block} cmake
151+
# CMake generated Testfile
152+
add_test([=[greeter_test]=] "C:/project/build/greeter.exe")
153+
set_tests_properties([=[greeter_test]=] PROPERTIES _BACKTRACE_TRIPLES "C:/project/CMakeLists.txt;34;add_test;C:/project/CMakeLists.txt;0;")
154+
subdirs("/.gtest")
155+
```
156+
157+
```{important}
158+
This file is generated during CMake configure and does not require the test executable to be built.
159+
It is only used by CTest to run the tests, like having a target to run the test executable.
160+
```
161+
162+
### Discovering tests with GoogleTest CTest integration
163+
164+
CMake provides a helpful module `GoogleTest` which simplifies test discovery and registration when using GTest:
165+
166+
```{code-block} cmake
167+
include(CTest)
168+
include(GoogleTest)
169+
170+
# Automatically discover all GoogleTest test cases in the greeter binary
171+
gtest_discover_tests(greeter
172+
# Optional: specify a WORKING_DIRECTORY or other properties
173+
)
174+
```
175+
176+
This approach removes the need to manually call add_test(NAME <some_name> COMMAND <test_executable>) for each test.
177+
Instead, `gtest_discover_tests` takes care of enumerating them by parsing the output of the actual test binary.
178+
179+
#### Details on how GoogleTest CTest integration works
180+
181+
When using `gtest_discover_tests`, CMake generates a `CTestTestfile.cmake` file that includes the tests discovered by GoogleTest.
182+
183+
After CMake configure (no tests executable are yet built nor executed) there is a `CTestTestfile.cmake` file that includes the tests discovered by GoogleTest.
184+
There is a `<component>_include.cmake` file generated for every component which is just defines a dummy test.
185+
186+
```{code-block} cmake
187+
:caption: greeeter[1]_include.cmake
188+
if(EXISTS "C:/build/greeter[1]_tests.cmake")
189+
include("C:/build/greeter[1]_tests.cmake")
190+
else()
191+
add_test(greeter_NOT_BUILT greeter_NOT_BUILT)
192+
endif()
193+
```
194+
195+
Executing `ctest` will fail because the command `greeter_NOT_BUILT` does not exist.
196+
197+
```{important}
198+
CMake will add a custom command (see `build.ninja` build file) to generate the `greeter_tests.cmake` file.
199+
This command will run the test executable and parse the output to generate the test cases.
200+
```
201+
202+
After building the test executable, the `greeeter[1]_tests.cmake` file is generated and this contains all tests discovered by GoogleTest.
203+
204+
```{code-block} cmake
205+
:caption: greeeter[1]_tests.cmake
206+
207+
add_test([=[GreeterTest.Greeting1]=] C:/project/build/greeter.exe [==[--gtest_filter=GreeterTest.Greeting1]==] --gtest_also_run_disabled_tests)
208+
set_tests_properties([=[GreeterTest.Greeting1]=] PROPERTIES WORKING_DIRECTORY C:/project/build SKIP_REGULAR_EXPRESSION [==[\[ SKIPPED \]]==])
209+
add_test([=[GreeterTest.Greeting2]=] C:/project/build/greeter.exe [==[--gtest_filter=GreeterTest.Greeting2]==] --gtest_also_run_disabled_tests)
210+
set_tests_properties([=[GreeterTest.Greeting2]=] PROPERTIES WORKING_DIRECTORY C:/project/build SKIP_REGULAR_EXPRESSION [==[\[ SKIPPED \]]==])
211+
set( greeter_TESTS GreeterTest.Greeting1 GreeterTest.Greeting2)
212+
```
213+
214+
Executing `ctest` will now run the tests and generate the JUnit report.
215+
216+
### VS Code integration
217+
218+
The CMake Tools extension for VS Code supports since version 1.14 a Test Explorer for CTest.
219+
The extension will automatically detect the CTest tests from the generated CTest files and display them in the Test Explorer.
220+
221+
This means that one can run, debug, and see the test results directly in the VS Code interface.
222+
223+
Of course, this approach inherits all the "limitations" of discovering the tests with CTest.
224+
After CMake configure, one will see only "\_NOT_BUILT" tests in the Test Explorer.
225+
226+
![NOT_BUILT tests displayed](images/not_build_tests.png)
227+
228+
Only after building the test executable, the tests will be visible.
229+
230+
![All tests displayed](images/all_tests.png)
231+
232+
```{important}
233+
When executing the tests from the Test Explorer, before running the tests, the CMake Tools extension will start the current select CMake build target.
234+
This might be confusing if one switches between build targets and suddenly when executing a test, a target is build which has nothing to do with the test.
235+
236+
I think their idea is to ensure that the test executable is built before running the tests, to update the test cases.
237+
```
238+
239+
When executing single tests, CTest is called with the `--tests-regex` option to run only the selected tests.
240+
The problem is that when a test fails there is no way to go from the Test Explorer to the failing test.
241+
This is a general issue with CTest that there is no automatic traceability between the test name and the test source file and line.
242+
There is another VS Code extension [CMake Test Explorer](https://marketplace.visualstudio.com/items?itemName=fredericbonnet.cmake-test-adapter)
243+
which attempts to address this issue by using the CTest properties feature. I did not test this extension. Maybe in a future blog post.
244+
245+
## Conclusion
246+
247+
Integrating the unit tests in the VS Code interface is a nice feature to have. CTest and GoogleTest provide a good way to discover and run the tests.
248+
While not perfect, the integration with the CMake Tools extension for VS Code is a good start.
249+
One can run, debug, and see the test results directly in the VS Code interface.
250+
251+
## References
252+
253+
- [Testing With CMake and CTest](https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html)
254+
- [VS Code CMake Tools Extension](https://github.com/microsoft/vscode-cmake-tools)

0 commit comments

Comments
 (0)