Overall Protocol for Testing

Before a new release can occur it must:

• Unit tests must all pass.
• Regression tests must be run using the benchmark dataset.
• Set output directory to '/tdata/CPAC/vversion number with no periods/output/benchmark.
• Set working directory to '/tdata/CPAC/vversion number with no periods/working/benchmark.
• Set crash directory to '/tdata/CPAC/vversion number with no periods/crash/benchmark.
• Set logs directory to '/tdata/CPAC/vversion number with no periods/logs/benchmark.
• Save the results for the last regression test for use in the benchmark package.
• Question: Could we run this in Docker using a modified Dockerfile for the C-PAC BIDS app?'
• Question: What will be our retention policy for benchmark dataset outputs? How long do we want to keep outputs from failed runs (up until the bug is fixed? longer in case a fix doesn't work as expected later on?) How long do we want to keep the outputs from previous versions of C-PAC on the test server?
• C-PAC must be run on novel datasets to determine if any anomalies occur with non-benchmark data:
• Perform a run on:
• Rhyme Judgement
• Serial Scanning
• Brain Genomics Superstruct Project
• Pipelines should be configured to use the following directories:
• Output directory : '/tdata/CPAC/vversion number with no periods/output/dataset name.
• Working directory : '/tdata/CPAC/vversion number with no periods/working/dataset name.
• Crash directory : '/tdata/CPAC/vversion number with no periods/crash/dataset name.
• Logs directory : '/tdata/CPAC/vversion number with no periods/logs/dataset name.
• GUI testing must be finished with no failures.
• All issues in the tracker must be iterated through and tested to ensure that they have been fixed.