Overall Protocol for Testing

This document outlines the testing process for before a new release of C-PAC can be made.

Preliminaries

• Nosetests must be run and all unit tests must pass.
• All fixes for issues on the kanban board must be tested to ensure that they have been fixed.

UI Testing

• GUI testing must be finished with no failures.
• Specific elements to look at:
  1. Subject list builder
  2. Pipeline configuration (individual-level)
  3. Group analysis
  4. Tool tips, in-app documentation and config comments - make sure they are current.
• This can be performed contemporaneously with a regression test run.

Regression Testing

• The regression test suite must be run using the benchmark dataset of 40 participants from ADHD200. The run will be done on on dozer natively, without virtualization.
• The output directory will be '/tdata/CPAC/vversion number with no periods/output/benchmark.
• The working directory will be '/tdata/CPAC/vversion number with no periods/working/benchmark.
• The crash directory will be '/tdata/CPAC/vversion number with no periods/crash/benchmark.
• The logs directory will be '/tdata/CPAC/vversion number with no periods/logs/benchmark.
• Retention policy:
• Save the results for the last regression test for use in the benchmark package and for comparison to the results of the next release (~ 333 GB for each set of outputs at present).
• There will be a maximum of 2 regression test runs stored on dozer at any given time (one for the last release, one for the release currently under development).
• Components of the Regression Test Suite:
• Group analysis.
• An ANTS run.
• An FSL run.
• Resource allocation
• Correlations are run on all of the above.
• A script kicks off the regression test run and runs correlations automatically, tarring the results and sending them via e-mail when then the run is complete.
• If all is well and we are ready to release, the output directory should be uploaded to AWS as the canonical benchmark results.

Installation Testing

• Perform a fresh installation on several platforms using the install script.
• Platforms:

1. CentOS 5 (Docker) - EOL : 2017-03-31
2. CentOS 6 (Docker) - EOL : 2020-11-30
3. CentOS 7 (Docker) - EOL : 2024-06-30
4. Ubuntu 12.04 LTS (Docker) - EOL : 2017-04-26
5. Ubuntu 14.04 LTS (Docker) - EOL : 2019-04
6. Ubuntu 16.04 LTS (Docker) - EOL : 2021-04
7. Ubuntu 16.10 (Docker) - EOL : 2017-07
8. OS X (10.6 is available on the C-PAC Mac)
9. Individual developer/tester laptops

Small Batch Testing

• Smaller scale tests of two participant sub-datasets are run on a variety of datasets / platforms. This is to ensure that runs execute successfully under diverse scenarios.
• A GitHub repository will contain a wide variety of pipeline configuration YAMLs to try out, with various options toggled on and off.
• Platforms (Linux distribution versions that are end-of-lifed will not be tested):

1. AWS : Perform a run on a freshly-formed AMI that tests the AWS functions (i.e., uses data stored in S3 for a participant list, puts data into S3 for outputs).
2. CentOS 5 (Docker) - EOL : 2017-03-31
3. CentOS 6 (Docker) - EOL : 2020-11-30
4. CentOS 7 (Docker) - EOL : 2024-06-30
5. Ubuntu 12.04 LTS (Docker) - EOL : 2017-04-26
6. Ubuntu 14.04 LTS (Docker) - EOL : 2019-04
7. Ubuntu 16.04 LTS (Docker) - EOL : 2021-04
8. Ubuntu 16.10 (Docker) - EOL : 2017-07
9. OS X (10.6 is available on the C-PAC Mac)
10. Individual developer/tester laptops

• Datasets

1. Rhyme Judgement
2. Serial Scanning
3. Brain Genomics Superstruct Project

• Pipelines should be configured to use the following directories on each platform:
• Output directory : '/tdata/CPAC/platform/vversion number with no periods/output/dataset name.
• Working directory : '/tdata/CPAC/platform/vversion number with no periods/working/dataset name.
• Crash directory : '/tdata/CPAC/platform/vversion number with no periods/crash/dataset name.
• Logs directory : '/tdata/CPAC/platform/vversion number with no periods/logs/dataset name.
• When these tests are run crash files and logs will be aggregated and examined before the final release. If anything needs to be fixed, regression testing will be re-run after C-PAC has been patched and the pipeline that produced the crash will be re-run.