PyTation

PyTation is a hybrid, fault-driven mutation testing framework for Python.
It introduces Python-specific mutation operators derived from empirical studies of real-world Python bugs and combines static and dynamic analysis to generate executable, non-trivial mutants.

PyTation is designed to evaluate the adequacy of existing Python test suites and to complement existing mutation testing tools such as Cosmic Ray.

Key Features

• Python-specific mutation operators grounded in empirical bug studies
• Hybrid static + dynamic analysis to ensure executable mutants
• Operator-level pruning heuristics to reduce equivalent and trivial mutants
• Execution-based mutant subsumption analysis
• Compatible with existing Python test suites (pytest-based)

List of Mutation Operators Implemented by PyTation

The mutation operators implemented in PyTation are:

Mutation Operator (Abbr. Name)	Before Mutation	After Mutation
1. Remove A Function Argument (RemFuncArg)	func(a₁, a₂, a₃, ...)	func(a₁, a₃, ...)
2. Remove Conversion Functions (RemConvFunc)	conv(a)	a
3. Remove Elements From Containers (RemElCont)	C = [e₁, ..., eᵢ, ..., eₙ]	[e₁, ..., eᵢ₋₁, eᵢ₊₁, ... eₙ]
4. Remove Expressions from Conditions (RemExpCond)	S = [e₁, ..., eᵢ, ..., eₙ]	S = [e₁, ..., eᵢ₋₁, eᵢ₊₁, ... eₙ]
5. Change Used Attribute (ChUsedAttr)	object.attr	object.new_attr
6. Remove Attribute Access (RemAttrAcc)	object.attr	object
7. Remove Method Call (RemMetCall)	object.method()	object

How to Run PyTation Over a Project:

Using Docker:

There is a Docker file available for PyTation. All of the benchmark projects are in benchmarks directory. To run PyTation on all projects using docker, run the following commands:

docker build -t pytation .
docker run -d --name pytation_ctr pytation

Then, you can run the following command to execute PyTation on one specific project:

docker exec -it pytation_ctr bash run_in_docker.sh <project_name>

For example:

docker exec -it pytation_ctr bash run_in_docker.sh blinker

This script will make a venv for the project, install the requirements, and run PyTation.

To run PyTation on all projects, you can use the following command:

docker exec -it pytation_ctr bash run_in_docker.sh

Note: This command will run PyTation on all projects, and it may take a long time.

Without Docker:

If you want to run PyTation on a specific project without using docker, you can follow the steps below:

1. Clone the project
2. Install PyTation on the project

pip install /path/to/pytation

3. Run the following command:

pytation -r <project_root> -i <include_paths> -e <exclude_paths>

For example:

pytation -r . -i src -e src/test

To use subset selection, you can use the following command:

pytation -r . -i src -ss operator_random_threshold

And to set timeout for test execution, you can use the following command:

pytation -r . -i src -ss operator_random_threshold -to 10

---

Acknowledgements

This work was supported in part by a Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant.