Skip to content

Latest commit

 

History

History
167 lines (118 loc) · 5.88 KB

testing.md

File metadata and controls

167 lines (118 loc) · 5.88 KB

Testing

Contents

Unit Tests

Crawl integrates the Catch2 unit testing framework. Tests are in the crawl-ref/source/catch2-tests directory. To compile and run all units tests, use:

make catch2-tests

Plug & Play / Bisect Testing

test_plug_and_play.cc is an optional source file for catch2 tests. If the source file is available, the make plug-and-play-tests command will use it to create a catch2 tests binary. If the source file is not available, the make plug-and-play-tests command will fail. This source file is in the .gitignore and should never be added to the git project.

The main use of test_plug_and_play.cc is for the file to be used in conjunction with the git bisect command. If you don't know how the git bisect command works, reference git bisect --help because otherwise none of the following will make sense.

test_plug_and_play.cc is a file which:

  1. Compilations of versions of crawl older than master can search for catch2 tests
  2. Will not be overwritten by git history changes, such as those which occur while using git bisect.

This combination of features allows for you to write tests once, then use them for any version of crawl which has previously existed, all the way back to the original implementation of catch2 support for crawl.

While you are hunting down a bug introduced in newer versions of crawl, see if you can create a catch2 test which checks for the bug in test_plug_and_play.cc. Verify this test works with make plug-and-play-tests. Then, see if this test works correctly when you git checkout a version of crawl which does not have the bug. If both are true, you can use test_plug_and_play.cc to easily find the bug.

Simply use git bisect as normally, but compile and run plug-and-play-tests instead of normal crawl. Whenever the bug is present, your test will fail and you can mark that commit as bad. Whenever the bug is absent, your test will pass and you can mark that commit as good.

This test process is so easy git bisect allows you to automate it. Look into git bisect run for more details.

When you've finished solving the bug, remember to move the new test from test_plug_and_play.cc to a regular test file included in the git project! This will help prevent regressions.

Functional (Lua) Tests

Crawl has a set of functional tests in the source/test/ directory, with each lua file being a separate unit test. To use unit tests, you must compile Crawl either with the macro DEBUG_DIAGNOSTICS defined (the debug make target) or with both the macros DEBUG and DEBUG_TESTS defined (the wizard make target, plus -DDEBUG_TESTS). You can then do

crawl -test

to run all unit tests. To run a particular test, you can do

crawl -test foo

and that will run all unit tests which have "foo" in their file name.

If you want to write your own unit tests, take a look at the files in source/test/ for examples. los_maps.lua and bounce.lua have examples which use vaults (maps) which are located in test/des. You might need to create new Lua functions in the source/l_*.cc files if none of the crawl.foo()/dgn.foo()/etc functions do what you need.

Arena Testing

You can use Crawl's arena mode to test a lot of things. See arena.txt for more information.

Code Coverage

Code coverage instrumentation is included in all debug & unit test builds. You can use it as follows:

  1. Compile the game in debug or unit test mode:
make debug
# or
make catch2-tests
  1. Run the compiled binary to generate coverage data:
./crawl -test
# or
# ./catch2-test-executable # no need, Makefile ran this automatically
  1. Generate the coverage report:
util/coverage # Add --help to see customisation options

Quirks

  1. Coverage data from multiple runs is combined. This can lead to confusing execution counts for source code lines in reports. Use make clean-coverage to remove all temporary coverage files (including reports).

A Quick Comparison of HTML Report Formats

Sub-expression coverage covers the granularity of testing a line of code like this:

if (test1() && test2())

lcov / genhtml

  • Included with GCC on many Linux distributions (if not, easy to install)
  • No sub-expression coverage (line-coverage only)

llvm-cov

  • Included with LLVM/Clang
  • Partial sub-expression coverage (will detect un-executed sub-expressions)

gcovr

  • Python 3-based utility, harder to install on mingw and older systems
  • Full sub-expression coverage (will detect if each sub-expression was tested with a positive and negative test case)

Implementation Details

Code coverage is handled very differently between the GCC and LLVM toolchains. The Makefile and util/coverage both auto-detect your toolchain, so this is abstracted away for general use. However, coverage percentages and details will be slightly different.

GCC

  1. Compile with -fprofile-arcs -ftest-coverage, this generates .gcno files (one per source file)
  2. When run, your executable creates .gcda (one per source file)
  3. Use either lcov/genhtml or gcovr:
    • lcov/genhtml
      1. Parse gcno/gcda files with lcov into a .info file
      2. Generate reports with genhtml.
    • gcovr
      1. Generate reports with gcovr.
        • (gcovr uses gcov to parse gcno/gcda files.)

LLVM

  1. Compile with -fprofile-instr-generate -fcoverage-mapping
  2. When run, your executable creates default.profraw
  3. Parse this with llvm-profdata into a .profdata file
  4. Generate reports with llvm-cov.