Skip to content

Commit 78b9c39

Browse files
bridadanbridadan
committed
Adding docs for the testing process for mbed OS 5
1 parent 8996227 commit 78b9c39

File tree

1 file changed

+187
-0
lines changed

1 file changed

+187
-0
lines changed

‎docs/testing_mbed_OS_5.md

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# Testing in mbed OS 5
2+
3+
The way tests are ran and compiled in mbed OS 5 is substantially different than in previous versions of mbed. Previously tests were located in one known location and a python file was kept (`tools/tests.py`) keeping track of their dependencies, capabilities, and configurations. mbed OS 5 has adopted a more distributed approach to testing. Now test code lives alongside the test code, and these are dynamically discovered by the test tools.
4+
5+
## Table of Contents
6+
7+
- [Using tests](#using-tests)
8+
- [Test code structure](#test-code-structure)
9+
- [Test discovery](#test-discovery)
10+
- [Test names](#test-names)
11+
- [Building tests](#building-tests)
12+
- [Running tests](#running-tests)
13+
- [Writing tests](#writing-tests)
14+
- [Debugging tests](#debugging-tests)
15+
- [Known issues](#known-issues)
16+
17+
## Test code structure
18+
19+
Tests can exist throughout mbed OS and your project's code. They are located under a special directory called `TESTS` (case is important!).
20+
21+
Placing code under this directory means it will be ignored when building applications and libraries. This code is only ever used when building tests. This is important since all tests require a `main()` function, and building it with your application would cause multiple `main()` functions to be defined.
22+
23+
In addition to being placed under a `TESTS` directory, test sources must exist under two other directories: a "test group" folder and a "test case" folder. The following is an example of this structure:
24+
```
25+
myproject/TESTS/test_group/test_case_1
26+
```
27+
28+
In this example, `myproject` is the project root and all the source files under the `test_case_1` directory will be included in the test. Any other source files from the OS, libraries, and your project that apply to your target's configuration will also be included in the build of your test.
29+
30+
### Test discovery
31+
32+
Since test cases can exist throughout a project, the tools must find them in your project's file structure before building them. This is done by searching for paths that match the pattern detailed above in the [Test code structure](#test-code-structure) section.
33+
34+
Test discovery also obeys the same rules that is used when building your project. This means that tests that are placed under a folder with a prefix like `TARGET_`, `TOOLCHAIN_`, or `FEATURE_` will only be discovered, built, and ran if your current configuration matches this prefix.
35+
36+
For example, if a test was placed under the folder `FEATURE_BLE` with the following path:
37+
38+
```
39+
myproject/mbed-os/features/FEATURE_BLE/TESTS/ble_tests/unit_test
40+
```
41+
42+
This test case would only be discovered if the target that was being testing supported the BLE feature. Otherwise, the test will be ignored.
43+
44+
Generally, a test should not be placed under a `TARGET_` or `TOOLCHAIN_` folder, since most tests should be designed to work for all target and toolchain configurations.
45+
46+
Tests can also be completely ignored by using the `.mbedignore` file descirbed [here]()
47+
48+
### Test names
49+
50+
A test case is named from its position in your project's file structure. For instance, in the above example, a test case with the path `myproject/TESTS/test_group/test_case_1` would be named `tests-test_group-test_case_1`. You will notice that the name is created by joining the directories that make up the path to the test case with a "dash" (`-`) character. This will be a unique name to identify the test case. You will see this name used throughout the build and testing process.
51+
52+
## Building tests
53+
54+
Tests can be built easily through mbed CLI. For information on using mbed CLI, please see its documentation.
55+
56+
When tests are built for a target and a given toolchain, the available tests are first discovered, then built in series. You can also create a "test specification" file, which can be used by our testing tools to run automated hardware tests. For more information on the test specification file, please see the docmentation [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
57+
58+
### Building process
59+
60+
The process for building tests is handled by the `test.py` script (not to be confused with `tests.py`) located under the `tools` directory. This handles the discovery and building of all tests cases for a given target and toolchain.
61+
62+
The actual process that occurs is as follows:
63+
64+
1. Build the "shared" code (all non test code), but do not link it
65+
- The resulting object files are placed in the build directory
66+
1. Find all tests that pertain to the given target and toolchain
67+
1. For each discovered test, build all of its source files and link it with the "shared" code that was built in step 1
68+
1. If specified, create a test specification file and place it in the given directory for use by testing tools
69+
- This is placed in the build directory by default when using mbed CLI
70+
71+
### Running tests
72+
73+
Automated tests can be ran easily through mbed CLI. For information on using mbed CLI, please see its documentation.
74+
75+
The testing process requires that the tests are built and that a test specification JSON file exists that describes these available tests. The test specification format is detailed [here](https://github.com/ARMmbed/greentea#test-specification-json-formatted-input).
76+
77+
The actual testing process is handled by the Greentea tool. To read more about this tool, please visit its [GitHub repository](https://github.com/ARMmbed/greentea).
78+
79+
### Writing tests
80+
81+
You can write your own tests for your project, or add more tests to mbed OS. Tests are written using the [Greentea client](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/greentea-client), [UNITY](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/unity), and [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest) frameworks, located in `/features/frameworks`. Below is an example test that uses all of these frameworks:
82+
83+
```c++
84+
#include "mbed.h"
85+
#include "greentea-client/test_env.h"
86+
#include "unity.h"
87+
#include "utest.h"
88+
#include "rtos.h"
89+
90+
using namespace utest::v1;
91+
92+
// A test that returns successfully is considered successful
93+
void test_success() {
94+
TEST_ASSERT(true);
95+
}
96+
97+
// Tests that assert are considered failing
98+
void test_failure() {
99+
TEST_ASSERT(false);
100+
}
101+
102+
utest::v1::status_t test_setup(const size_t number_of_cases) {
103+
// Setup Greentea using a reasonable timeout in seconds
104+
GREENTEA_SETUP(40, "default_auto");
105+
return verbose_test_setup_handler(number_of_cases);
106+
}
107+
108+
// Test cases
109+
Case cases[] = {
110+
Case("Testing success test", test_success),
111+
Case("Testing failure test", test_failure),
112+
};
113+
114+
Specification specification(test_setup, cases);
115+
116+
// Entry point into the tests
117+
int main() {
118+
return !Harness::run(specification);
119+
}
120+
```
121+
122+
This test will first run a case that succeeds, then it will run a test that fails. This is a good template to use when creating tests. For more complex testing examples, please see the documentation for [utest](https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/utest).
123+
124+
## Debugging tests
125+
126+
Debugging tests is a crucial part of the development and porting process. This section will cover exporting the test, then driving the test with the test tools while the target is attached to a debugger.
127+
128+
### Exporting tests
129+
130+
Currently, the easiest way to export a test is to copy the test's source code from its test folder to your project's root. This way it will be treated like a normal application by the tools.
131+
132+
You can find the path to the test you wish to export by running the following command:
133+
134+
```
135+
mbed test --compile-list -n <test name>
136+
```
137+
138+
Once you've copied all of the test's source files to your project root, go ahead and export your project:
139+
140+
```
141+
mbed export -i <ide name>
142+
```
143+
144+
Your exported project should now be in `projectfiles/<ide>_<target>`. Go ahead and open this project in your IDE.
145+
146+
### Running a test while debugging
147+
148+
Assuming your test has exported correctly to your IDE, go ahead and build the project and load it onto your target via your debugger.
149+
150+
Bring the target out of reset and run the program. Your target will now be waiting for a synchronizing character string to be sent from the test tools over the serial port. The `mbed test` command cannot be run as that will attempt to flash the device, which you've already done with your IDE.
151+
152+
Instead, the underlying test tools can be used to drive the test. [htrun](https://github.com/ARMmbed/htrun) is the tool that needs to be used in this case. This is installed when you install the requirements for mbed OS. However, if you don't have it installed you can do this by running `pip install mbed-host-tests`.
153+
154+
First, find your target's serial port by running the following command:
155+
156+
```
157+
$ mbed detect
158+
159+
[mbed] Detected KL46Z, port COM270, mounted D:
160+
161+
...
162+
```
163+
164+
From the output, take note of where your target's serial port (in this case, it's `COM270`).
165+
166+
Run the following command when your device is running the test in your debugger:
167+
168+
```
169+
mbedhtrun --skip-flashing --skip-reset -p <serial port>:9600
170+
```
171+
172+
Replace `<serial port>` with the serial port you found by running `mbed detect` above.
173+
174+
So for the example above, the command would be:
175+
176+
```
177+
mbedhtrun --skip-flashing --skip-reset -p COM270:9600
178+
```
179+
180+
This will detect your attached target and drive the test. At this point the test should proceed and you should be able to debug it. If you need to rerun the test, simply flash and/or reset the device with your debugger, run the program, and run the same command.
181+
182+
For an explanation of the arguments used in this command, please run `mbedhtrun --help`.
183+
184+
## Known issues
185+
186+
- There cannot be a `main()` function outside of a `TESTS` directory when building and running tests. This is because this function will be included in the "shared" build as described in the [Building process](#building-process) section. When the test code is compiled and linked with this "shared" build, a linker error will occur due to their being multiple `main()` functions defined. For this reason, please either rename your main application file if you need to build and run tests or use a different project.
187+
- **NOTE:** This does not affect building projects or applications, just building and running tests.

0 commit comments

Comments
 (0)