Skip to content

Commit 82480fd

Browse files
jamesmunnsjamesmunns
authored
Merge pull request #188 from jamesmunns/rust-ci
Rust Based Regression Tester
2 parents 631ab3e + 0554381 commit 82480fd

File tree

9 files changed

+4568
-0
lines changed

9 files changed

+4568
-0
lines changed

‎.gitlab-ci.yml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
build:stable:
2+
image: rust:stretch
3+
script:
4+
- cargo build
5+
6+
build:nightly:
7+
image: rustlang/rust:nightly
8+
script:
9+
- cargo build
10+
11+
regress:nightly:
12+
image: rustlang/rust:nightly
13+
script:
14+
- cargo build --release
15+
- cd ci/svd2rust-regress
16+
- rm -rf ./output
17+
- cargo run --release -- --long-test

‎README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,12 @@
99

1010
# [API](https://docs.rs/svd2rust)
1111

12+
# Testing Locally
13+
14+
`svd2rust-regress` is a helper program for regression testing changes against `svd2rust`. This tool can be used locally to check modifications of `svd2rust` locally before submitting a PR.
15+
16+
Check out the [svd2rust-regress README](ci/svd2rust-regress/README.md) for information on how to use this tool.
17+
1218
## License
1319

1420
Licensed under either of

‎ci/svd2rust-regress/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
output/

‎ci/svd2rust-regress/Cargo.toml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
[package]
2+
name = "svd2rust-regress"
3+
version = "0.1.0"
4+
authors = ["James Munns <[email protected]>"]
5+
6+
[dependencies]
7+
reqwest = "0.8"
8+
rayon = "1.0"
9+
structopt = "0.2"
10+
error-chain = "0.11"
11+
inflections = "1.1.0"

‎ci/svd2rust-regress/README.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# `svd2rust` Regression Tester
2+
3+
`svd2rust-regress` is a helper program for regression testing changes against `svd2rust`. It uses `rayon` to parallelize testing of multiple chips simultaneously.
4+
5+
## What it does
6+
7+
`svd2rust-regress` will do the following things for each svd/chip tested:
8+
9+
1. Create a new crate for that chip in `output/<chip>`, populated with the architecture specific dependencies
10+
2. Download the `.svd` file for that chip
11+
3. Run `svd2rust` to generate `output/<chip>/src/lib.rs`
12+
4. Run `cargo check` to ensure the project still builds
13+
14+
## Usage
15+
16+
### Preconditions
17+
18+
By default, `svd2rust-regress` assumes you have already built `svd2rust` in the root of this repository in `--release` mode. If this is not possible, it is possible to specify the path to an `svd2rust` binary (see **Options** below).
19+
20+
### Output
21+
22+
For each test case, `svd2rust-regress` will output the result.
23+
24+
Pass results look like this:
25+
26+
```text
27+
Passed: spansion_mb9af12x_k - 23 seconds
28+
```
29+
30+
Fail results look like this:
31+
32+
```text
33+
Failed: si_five_e310x - 0 seconds - Error(Msg("Process Failed! - cargo check"), State { next_error: None, backtrace: None })
34+
```
35+
36+
If all test cases passed, the return code will be `0`. If any test cases failed, the return code will be `1`.
37+
38+
### Options
39+
40+
Here are the options for running `svd2rust-regress`:
41+
42+
43+
```text
44+
svd2rust-regress 0.1.0
45+
James Munns <[email protected]>
46+
47+
USAGE:
48+
svd2rust-regress [FLAGS] [OPTIONS]
49+
50+
FLAGS:
51+
-b, --bad-tests Include tests expected to fail (will cause a non-zero return code)
52+
-h, --help Prints help information
53+
-l, --long-test Run a long test (it's very long)
54+
-V, --version Prints version information
55+
56+
OPTIONS:
57+
-a, --architecture <arch> Filter by architecture, case sensitive, may be combined with other filters Options
58+
are: "CortexM", "RiscV", and "Msp430"
59+
-p, --svd2rust-path <bin_path> Path to an `svd2rust` binary, relative or absolute. Defaults to
60+
`target/release/svd2rust[.exe]` of this repository (which must be already built)
61+
-c, --chip <chip> Filter by chip name, case sensitive, may be combined with other filters
62+
-m, --manufacturer <mfgr> Filter by manufacturer, case sensitive, may be combined with other filters
63+
```
64+
65+
### Filters
66+
67+
`svd2rust-regress` allows you to filter which tests will be run. These filters can be combined (but not repeated).
68+
69+
For example, to run all `RiscV` tests:
70+
71+
```bash
72+
# in the ci/svd2rust-regress folder
73+
cargo run --release -- -a RiscV
74+
Finished release [optimized] target(s) in 0.0 secs
75+
Running `target/release/svd2rust-regress -a RiscV`
76+
Passed: si_five_e310x - 7 seconds
77+
```
78+
79+
To run against any chip named `MB9AF12xK`:
80+
81+
```bash
82+
cargo run --release -- --long-test -c MB9AF12xK
83+
Finished release [optimized] target(s) in 0.0 secs
84+
Running `target/release/svd2rust-regress --long-test -c MB9AF12xK`
85+
Passed: spansion_mb9af12x_k - 23 seconds
86+
Passed: fujitsu_mb9af12x_k - 25 seconds
87+
```
88+
89+
To run against specifically the `Fujitsu` `MB9AF12xK`:
90+
```bash
91+
cargo run --release -- --long-test -c MB9AF12xK -m Fujitsu
92+
Finished release [optimized] target(s) in 0.0 secs
93+
Running `target/release/svd2rust-regress --long-test -c MB9AF12xK -m Fujitsu`
94+
Passed: fujitsu_mb9af12x_k - 19 seconds
95+
```

‎ci/svd2rust-regress/src/errors.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
error_chain!{}

‎ci/svd2rust-regress/src/main.rs

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
#[macro_use]
2+
extern crate error_chain;
3+
extern crate inflections;
4+
extern crate rayon;
5+
extern crate reqwest;
6+
#[macro_use]
7+
extern crate structopt;
8+
9+
mod tests;
10+
mod errors;
11+
mod svd_test;
12+
13+
use std::path::PathBuf;
14+
use structopt::StructOpt;
15+
use rayon::prelude::*;
16+
use std::sync::atomic::{AtomicBool, Ordering};
17+
use std::process::exit;
18+
use std::time::Instant;
19+
20+
#[derive(StructOpt, Debug)]
21+
#[structopt(name = "svd2rust-regress")]
22+
struct Opt {
23+
/// Run a long test (it's very long)
24+
#[structopt(short = "l", long = "long-test")]
25+
long_test: bool,
26+
27+
/// Path to an `svd2rust` binary, relative or absolute.
28+
/// Defaults to `target/release/svd2rust[.exe]` of this repository
29+
/// (which must be already built)
30+
#[structopt(short = "p", long = "svd2rust-path", parse(from_os_str))]
31+
bin_path: Option<PathBuf>,
32+
33+
/// Filter by chip name, case sensitive, may be combined with other filters
34+
#[structopt(short = "c", long = "chip")]
35+
chip: Option<String>,
36+
37+
/// Filter by manufacturer, case sensitive, may be combined with other filters
38+
#[structopt(short = "m", long = "manufacturer")]
39+
mfgr: Option<String>,
40+
41+
/// Filter by architecture, case sensitive, may be combined with other filters
42+
/// Options are: "CortexM", "RiscV", and "Msp430"
43+
#[structopt(short = "a", long = "architecture")]
44+
arch: Option<String>,
45+
46+
/// Include tests expected to fail (will cause a non-zero return code)
47+
#[structopt(short = "b", long = "bad-tests")]
48+
bad_tests: bool,
49+
50+
// TODO: Specify smaller subset of tests? Maybe with tags?
51+
// TODO: Early fail
52+
// TODO: Compile svd2rust?
53+
}
54+
55+
/// Validate any assumptions made by this program
56+
fn validate_tests(tests: &[&tests::TestCase]) {
57+
use std::collections::HashSet;
58+
59+
let mut fail = false;
60+
61+
// CONDITION 1: All mfgr+chip names must be unique
62+
let mut uniq = HashSet::new();
63+
for t in tests {
64+
let name = t.name();
65+
if !uniq.insert(name.clone()) {
66+
fail = true;
67+
eprintln!("{} is not unique!", name);
68+
}
69+
}
70+
71+
if fail {
72+
panic!("Tests failed validation");
73+
}
74+
}
75+
76+
fn main() {
77+
let opt = Opt::from_args();
78+
79+
// Validate all test pre-conditions
80+
validate_tests(tests::TESTS);
81+
82+
// Determine default svd2rust path
83+
let default_svd2rust_iter = ["..", "..", "..", "..", "target", "release"];
84+
85+
let default_svd2rust = if cfg!(windows) {
86+
default_svd2rust_iter.iter().chain(["svd2rust.exe"].iter())
87+
} else {
88+
default_svd2rust_iter.iter().chain(["svd2rust"].iter())
89+
}.collect();
90+
91+
let bin_path = match opt.bin_path {
92+
Some(ref bp) => bp,
93+
None => &default_svd2rust,
94+
};
95+
96+
// collect enabled tests
97+
let tests = tests::TESTS
98+
.iter()
99+
// Short test?
100+
.filter(|t| t.should_run(!opt.long_test))
101+
// selected architecture?
102+
.filter(|t| {
103+
if let Some(ref arch) = opt.arch {
104+
arch == &format!("{:?}", t.arch)
105+
} else {
106+
true
107+
}
108+
})
109+
// selected manufacturer?
110+
.filter(|t| {
111+
if let Some(ref mfgr) = opt.mfgr {
112+
mfgr == &format!("{:?}", t.mfgr)
113+
} else {
114+
true
115+
}
116+
})
117+
// Specify chip - note: may match multiple
118+
.filter(|t| {
119+
if let Some(ref chip) = opt.chip {
120+
chip == t.chip
121+
} else {
122+
true
123+
}
124+
})
125+
// Run failable tests?
126+
.filter(|t| opt.bad_tests || t.should_pass)
127+
.collect::<Vec<_>>();
128+
129+
let any_fails = AtomicBool::new(false);
130+
131+
// TODO: It would be more efficient to reuse directories, so we don't
132+
// have to rebuild all the deps crates
133+
tests.par_iter().for_each(|t| {
134+
let start = Instant::now();
135+
136+
match svd_test::test(t, &bin_path) {
137+
Ok(()) => {
138+
eprintln!(
139+
"Passed: {} - {} seconds",
140+
t.name(),
141+
start.elapsed().as_secs()
142+
);
143+
}
144+
Err(e) => {
145+
any_fails.store(true, Ordering::Release);
146+
eprintln!(
147+
"Failed: {} - {} seconds - {:?}",
148+
t.name(),
149+
start.elapsed().as_secs(),
150+
e
151+
);
152+
}
153+
}
154+
});
155+
156+
if any_fails.load(Ordering::Acquire) {
157+
exit(1);
158+
} else {
159+
exit(0);
160+
}
161+
}

0 commit comments

Comments
 (0)