Skip to content

Commit b3b5632

Browse files
bakaqbakaq
authored
Merge pull request bakaq#39 from bakaq/testing-improvements-3
Testing improvements (Part 3): Snapshot testing
2 parents bed2e8c + 1d923cf commit b3b5632

File tree

10 files changed

+227
-6
lines changed

10 files changed

+227
-6
lines changed

‎CONTRIBUTING.md‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,49 @@
1-
# Development environment
1+
## Development environment
22

3-
## Nix
3+
### Nix
44

55
This repository has a nix flake that has everything you will need to run the tests.
66
You can just use `nix develop` or [`direnv`](https://direnv.net/) with the provided `.envrc`.
77
Notice that it uses a newer version of Scryer Prolog from the git repo than the one provided by
88
nixpkgs, so you may need to compile it, which may take some time. You can use the same binary
99
cache as the CI with `cachix use pkg-pl` to get the precompiled environment.
1010

11-
## Non-Nix
11+
### Non-Nix
1212

1313
You will need a [Scryer Prolog](https://github.com/mthom/scryer-prolog) built close to `master`
1414
(`0.9.4`, the stable release, is too old for what we are doing here),
1515
[just](https://github.com/casey/just) and git.
1616

17-
# Running the tests
17+
## Running the tests
18+
19+
You can run `just test` in the root directory and it will run all the tests, same as CI. You can
20+
also run `just ci` to run the entire CI.
21+
22+
## Writing tests
23+
24+
### Snapshot tests
25+
26+
The snapshot tests are in the `tests/snapshot/` directory. The idea is that we run an arbitrary
27+
script and check that everything it does what we expect.
28+
29+
A test case is composed of the following files in the `tests/snapshot/cases/` directory:
30+
31+
- The test script `test_name.sh` that runs the test.
32+
- (Optional) A `test_name.stdin` file whose contents will be passed to the test script in stdin.
33+
- (Optional) A `test_name.in` directory where the script will be run. Can be used to provide files
34+
for the test to operate on.
35+
- (Can be generated) A `test_name.stdout` with the expected stdout of the test.
36+
- (Can be generated) A `test_name.stderr` with the expected stderr of the test.
37+
- (Can be generated) A `test_name.status` with the expected exit status code of the test.
38+
39+
The last 3 files can be generated from the current behavior of the test with `just snapshot`.
40+
41+
In the `tests/snapshot/` directory, you can use the following just commands to deal with snapshot
42+
tests:
43+
44+
- `just test` actually runs and checks the tests.
45+
- `just dump` runs the tests and puts the generated files in a `dump/` directory for inspection
46+
without checking them. Useful for manual inspection.
47+
- `just snapshot` runs the tests and puts the generated files in the `cases/` directory.
48+
Useful if you are pretty sure the current behavior is correct.
1849

19-
You can run `just test` in the root directory and it will run all the tests, same as CI.

‎justfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,4 +53,5 @@ test: build
5353
# Cleans everything that is generated during builds or tests
5454
clean:
5555
just example/clean
56+
just tests/clean
5657
rm -f bakage.pl.gen

‎tests/justfile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
test: integration-tests
1+
test: snapshot-tests integration-tests
22

33
integration-tests:
44
#!/bin/sh
@@ -16,3 +16,9 @@ integration-tests:
1616
done
1717

1818
exit "$exit_code"
19+
20+
snapshot-tests:
21+
just snapshot/test
22+
23+
clean:
24+
just snapshot/clean

‎tests/snapshot/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
dump/
2+
tmp_snapshot/

‎tests/snapshot/cases/cli_help.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/sh
2+
3+
bakage() {
4+
../../bakage.pl "$@"
5+
}
6+
7+
echo "=== Testing help with various different color configurations ==="
8+
bakage --help --color always
9+
echo "================================================================"
10+
bakage --help --color never
11+
echo "================================================================"
12+
CLICOLOR_FORCE=1 bakage --help
13+
echo "================================================================"
14+
NO_COLOR=1 bakage --help
15+
16+
echo "=== Install help ==="
17+
bakage install --help --color always
18+
echo "================================================================"
19+
bakage install --help --color never
20+
echo "================================================================"

‎tests/snapshot/cases/cli_help.status‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
0

‎tests/snapshot/cases/cli_help.stderr‎

Whitespace-only changes.

‎tests/snapshot/cases/cli_help.stdout‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
=== Testing help with various different color configurations ===
2+
Bakage: an experimental package manager for Prolog
3+
4+
Usage: bakage [OPTIONS] [COMMAND]
5+
6+
Options:
7+
--color When to use color. Valid options: auto, always, never
8+
-h, --help Print help
9+
10+
Commands:
11+
install Installs the dependencies of the current package
12+
================================================================
13+
Bakage: an experimental package manager for Prolog
14+
15+
Usage: bakage [OPTIONS] [COMMAND]
16+
17+
Options:
18+
--color When to use color. Valid options: auto, always, never
19+
-h, --help Print help
20+
21+
Commands:
22+
install Installs the dependencies of the current package
23+
================================================================
24+
Bakage: an experimental package manager for Prolog
25+
26+
Usage: bakage [OPTIONS] [COMMAND]
27+
28+
Options:
29+
--color When to use color. Valid options: auto, always, never
30+
-h, --help Print help
31+
32+
Commands:
33+
install Installs the dependencies of the current package
34+
================================================================
35+
Bakage: an experimental package manager for Prolog
36+
37+
Usage: bakage [OPTIONS] [COMMAND]
38+
39+
Options:
40+
--color When to use color. Valid options: auto, always, never
41+
-h, --help Print help
42+
43+
Commands:
44+
install Installs the dependencies of the current package
45+
=== Install help ===
46+
Install package dependencies
47+
48+
Usage: bakage install [OPTIONS]
49+
50+
Options:
51+
--color When to use color. Valid options: auto, always, never
52+
-h, --help Print help
53+
================================================================
54+
Install package dependencies
55+
56+
Usage: bakage install [OPTIONS]
57+
58+
Options:
59+
--color When to use color. Valid options: auto, always, never
60+
-h, --help Print help
61+
================================================================

‎tests/snapshot/justfile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
test:
2+
./snapshot.sh test
3+
4+
dump:
5+
./snapshot.sh dump
6+
7+
snapshot:
8+
./snapshot.sh snapshot
9+
10+
clean:
11+
rm -rf dump

‎tests/snapshot/snapshot.sh‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/bin/sh
2+
3+
set -eu
4+
5+
snapshot_test() {
6+
test_file=$(realpath "$1")
7+
test_name=$(basename "$test_file" ".sh")
8+
9+
printf 'Snapshoting test "%s"\n' "$test_name"
10+
11+
if [ -f "cases/${test_name}.stdin" ]; then
12+
stdin_file=$(realpath "cases/${test_name}.stdin")
13+
else
14+
stdin_file="/dev/null"
15+
fi
16+
17+
stdout_file=$(realpath "tmp_snapshot/${test_name}.stdout")
18+
stderr_file=$(realpath "tmp_snapshot/${test_name}.stderr")
19+
status_file=$(realpath "tmp_snapshot/${test_name}.status")
20+
21+
original_pwd=$(pwd)
22+
has_in_dir=false
23+
if [ -d "cases/${test_name}.in" ]; then
24+
has_in_dir=true
25+
cp -r "cases/${test_name}.in" "cases/${test_name}.in.bak"
26+
cd "cases/${test_name}.in"
27+
fi
28+
29+
status_code=0
30+
sh "$test_file" \
31+
< "$stdin_file" \
32+
> "$stdout_file" \
33+
2> "$stderr_file" \
34+
|| status_code="$?"
35+
printf '%s' "$status_code" > "$status_file"
36+
37+
if [ "$has_in_dir" = true ]; then
38+
cd "$original_pwd"
39+
rm -rf "cases/${test_name}.in"
40+
mv "cases/${test_name}.in.bak" "cases/${test_name}.in"
41+
fi
42+
}
43+
44+
make_snapshot() {
45+
mkdir -p tmp_snapshot
46+
for test_file in cases/*.sh; do
47+
snapshot_test "$test_file"
48+
done
49+
}
50+
51+
diff_test() {
52+
test_name="$1"
53+
54+
printf 'Diffing test "%s"\n' "$test_name"
55+
56+
status_code=0
57+
for suffix in stdout stderr status; do
58+
diff --unified "cases/${test_name}.${suffix}" "tmp_snapshot/${test_name}.${suffix}" \
59+
|| status_code=1
60+
done
61+
return "$status_code"
62+
}
63+
64+
case "$1" in
65+
dump)
66+
make_snapshot
67+
rm -rf dump
68+
mv tmp_snapshot dump
69+
;;
70+
snapshot)
71+
make_snapshot
72+
mv tmp_snapshot/* .
73+
rm -rf tmp_snapshot
74+
;;
75+
test)
76+
make_snapshot
77+
overall_status_code=0
78+
for test_file in cases/*.sh; do
79+
test_name=$(basename "$test_file" ".sh")
80+
diff_test "$test_name" || overall_status_code=1
81+
done
82+
rm -rf tmp_snapshot
83+
exit "$overall_status_code"
84+
;;
85+
*)
86+
printf '%s\n' "Unknown command"
87+
exit 1
88+
;;
89+
esac

0 commit comments

Comments
 (0)