Regression Testing

Overview

Regression tests are comparisons of code output to accepted solutions. They are used to ensure that parts of the code that work are not broken in the course of development. There should ultimately be a suite of stringent tests utilizing different parts of the code. Modifications to the code should not be pushed to the repository if they break regression tests.

All regression testing is confined to athena/test/regression/; running these tests should have no effect outside this directory.

The regression tests for Athena++ are written in Python. They should be compatible with any Python 2.7 distribution. If they break in later versions of Python, please file a bug report and we can try to fix any incompatibilities.

Usage

From the top-level athena/ directory go to the regression test directory:

cd test/regression

To run all test scripts that have been written, simply call

python run_tests.py

Instead of running all tests, a single test can be run with

python run_tests.py -s <test_name>

where the desired test is implemented in scripts/tests/<test_name>.py.

Writing new tests

This section is written from the perspective of the athena/test/regression/ directory.

Test locations

In order to create a new regression test, start with a new file <test_name>.py in scripts/tests/. No other file need be modified; in particular, the test does not need to be enrolled anywhere. In fact, the top-level script that is called, run_tests.py, should not be modified.

A new test may very well need a new saved dataset to compare against. All such data files are kept in data/. New files, usually Athena++ outputs, can be added here if need be. Data files in this directory are considered part of the repository: new files should be committed, and other developers' files should not be tampered with. Please refrain from including excessively large and unwieldy files.

One may want to write common functions for use in other tests. These can be placed in scripts/utils/ as detailed below.

Definition of a test

The file <test_name>.py is a valid test if it defines a valid Python function run_tests() that takes no inputs and returns either True or False depending on whether the test passes or not.

There are no other requirements for a test to work, but of course tests should behave themselves and should not modify/delete anything that could affect the rest of the code.

See a preexisting file, such as gr_shock_tube.py, for a working example.

Side effects

Running regression tests should leave the files outside the regression directory unchanged. This is tricky, since the configure script overwrites athena/Makefile and athena/src/defs.hpp. To work around this, the main regression script run_tests.py (which should not be modified) saves copies of these files, should they exist, before running any tests. Upon completing tests or encountering an exception, the regression script writes the original files back (or deletes whatever is there if those files did not exist before).

Because calling make on the Makefile output by the configure script would overwrite the objects and executable in the main directory, one must be careful in calling make. Instead of calling it directly, the make() function in scripts/utils/athena.py overrides the definitions in the Makefile to place the executable and all objects inside the regression directory. The executable will then place its outputs in the same location.

Utilities

Because most regression tests will perform similar actions, a set of common utility functions is provided in scripts/utils/.

To use these functions in a script, one must import the appropriate file. For example, from the test implementation in scripts/tests/<test_name>.py, one can call the make() function in scripts/utils/athena.py by writing

import scripts.utils.athena as athena
athena.make()

Athena++ interfacing

The file scripts/utils/athena.py defines Python routines for configuring, compiling, and running Athena++, as well as reading in certain output files. The publicly useful functions are detailed here.

configure(*args, **kwargs)

• Inputs:
  • Positional: any number of strings, to be prefaced with - when calling athena/configure.py; must go before keywords
  • Keywords: any number of key='val' assignments, to be used as --key=val when calling athena/configure.py; must go after positional arguments
• Outputs: (none)
• Example: configure('omp', prob='shock_tube') will result in the code being configured as though one called python configure.py -omp --prob=shock_tube from the home directory.

make()

• Inputs: (none)
• Outputs: (none)
• Example: Simply call make() and the code will be compiled into obj/ and linked into bin/. Note these are the subdirectories of athena/test/regression/, not those of the same names in athena/.

run(input_filename, arguments)

• Inputs:
  • input_filename: string containing the path to the desired input file, relative to athena/inputs/
  • arguments: list of strings, one for each command-line argument to be fed to Athena++
• Outputs: (none)
• Example: The following code snippet runs Athena++ with a relativistic shock tube, ensuring the CFL number, simulation time, and number of grid points are correct:

import scripts.utils.athena as athena
arguments = ['time/cfl_number=0.4', 'time/tlim=0.4', 'mesh/nx1=400']
athena.run('hydro_sr/athinput.mb_1', arguments)

read(filename, headings=None)

• Inputs:
  • filename: string containing the path to the desired Athena++ 1D tab output file to be read, relative to athena/test/regression
  • headings (optional, default None): list of strings to name columns of output file, excluding first column (zone number)
• Outputs: If headings is None, a 2D numpy array containing the data from the file, excluding the first column. Otherwise a dictionary whose keywords are the elements in headings and whose values are the corresponding 1D arrays read from the file.
• Example: Suppose one wants to read athena/test/regression/data/my_output.tab. The following code snippet does this, printing the values from the third column in the file:

import scripts.utils.athena as athena
data = athena.read('data/my_output.tab', ['x', 'rho', 'pgas', 'v1', 'v2', 'v3'])
print(data['rho'])

Comparisons between datasets

TODO: write