You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/side_quests/nf-test.md
+29-36Lines changed: 29 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,14 @@
1
1
# Side Quest: nf-test
2
2
3
-
Being able to systematically test that every part of your workflow is doing what it's supposed to do is critical for reproducibility and long-term maintenance.
4
-
And it's also helpful during the development process!
3
+
Being able to systematically test that every part of your workflow is doing what it's supposed to do is critical for reproducibility and long-term maintenance, and can be a huge help during the development process.
5
4
6
-
This is (hopefully) not controversial, but let's take a minute to talk about why testing is so important.
5
+
Let's take a minute to talk about why testing is so important. If you're developing a workflow, one of the first things you will do is grab some test data that you know is valid and should produce a result. You add the first process to the pipeline and wire it up to your inputs to make it work. Then, to check it's all working, so you run it on the test data. Assuming that works, you move on to the next process and run the test data again. You repeat this process until you have a pipeline that you're happy with.
7
6
8
-
Let's imagine you're developing a workflow. One of the first things you will do is grab some test data that you know is valid and should produce a result. You add the first process to the pipeline and wire it up to your inputs to make it work. Then, to check it's all working, you run it on the test data. Assuming it works, you move on to the next process and run the test data again. You repeat this process until you have a pipeline that you're happy with.
9
-
10
-
Maybe you add a simple true or false parameter such as `--skip_process`, now you must run the pipeline twice, once with each parameter to make sure it works as expected. But wait, how do we check if the `--skip_process` actually skips the process? We have to dig into the outputs or check the log files! This is a pain and prone to error.
7
+
Then, maybe you add a simple true or false parameter such as `--skip_process`. Now you must run the pipeline twice, once with each parameter to make sure it works as expected. But wait, how do we check if the `--skip_process` actually skips the process? We have to dig into the outputs or check the log files! This is a pain and prone to error.
11
8
12
9
As you develop your pipeline, it will quickly become so complex that manually testing every iteration is slow and error prone. Furthermore, if you do find an error it will be very difficult to pin down exactly where in your pipeline the error is coming from. This is where testing comes in.
13
10
14
-
Testing allows you to systematically check that every part of your pipeline is working as expected. The benefits to a developer are huge:
11
+
Testing allows you to systematically check that every part of your pipeline is working as expected. The benefits to a developer of well written tests are huge:
15
12
16
13
-**Confidence**: Because the tests cover the entire pipeline, you can be be confident changing something doesn't affect anything else
17
14
-**Trust**: When multiple developers work on the pipeline, they know the other developers haven't broken the pipeline and every component.
@@ -23,9 +20,10 @@ There are lots of different types of tests we can write:
23
20
1.**Module-level tests**: For individual processes
24
21
2.**Workflow-level tests**: For a single workflow
25
22
3.**Pipeline-level tests**: For the pipeline as a whole
26
-
4.**Integration tests**: For the interaction of the pipeline with other systems
27
-
5.**Performance tests**: For the speed and efficiency of the pipeline
28
-
6.**Stress tests**: To identify the limits of the pipeline
23
+
4.**Performance tests**: For the speed and efficiency of the pipeline
24
+
5.**Stress tests**: To identify the limits of the pipeline
25
+
26
+
Testing individual processes is analagous to unit tests in other languages. Testing the workflow or the entire pipeline is analagous to what's called integration tests in other languages, where we test the interactions of the components.
29
27
30
28
[**nf-test**](https://www.nf-test.com/) is a tool that allows you to write module, workflow and pipeline level test. In short, it allows you to systematically check every individual part of the pipeline is working as expected, _in isolation_.
31
29
@@ -60,6 +58,16 @@ Let's run the workflow to make sure it's working as expected.
"Wait, what? I just ran the workflow and it worked! How is that a test?"
@@ -165,9 +173,14 @@ The `test` block is the actual test. It contains the following:
165
173
-`when`: The conditions under which the test should be run. This includes the parameters that will be used to run the pipeline.
166
174
-`then`: The assertions that should be made. This includes the expected outcomes of the pipeline.
167
175
176
+
In plain English, the logic of the test reads as follows:
177
+
"**When** these _parameters_ are provided to this _pipeline_, **then** we expect to see these results."
178
+
179
+
This isn't a functional test, we will demonstrate how to turn it into one in the next section.
180
+
168
181
### A Note on Test Names
169
182
170
-
In the example above, we used the name "Should run without failures" which is appropriate for a basic test that just checks if the pipeline runs successfully. However, as we add more specific test cases, we should use more descriptive names that indicate what we're actually testing. For example:
183
+
In the example above, we used the default name "Should run without failures" which is appropriate for a basic test that just checks if the pipeline runs successfully. However, as we add more specific test cases, we should use more descriptive names that indicate what we're actually testing. For example:
171
184
172
185
- "Should convert input to uppercase" - when testing specific functionality
As we add more assertions and specific test cases later, we'll use these more descriptive names to make it clear what each test is verifying.
184
197
185
-
In plain English, the logic of the test reads as follows:
186
-
"**When** these _parameters_ are provided to this _pipeline_, **then** we expect to see these results."
187
-
188
-
This isn't a functional test, we will demonstrate how to turn it into one in the next section.
189
-
190
198
## 1.2. Run the test
191
199
192
200
Let's run the test to see what happens.
@@ -407,9 +415,7 @@ Success! The pipeline runs successfully and the test passes. Now we have began t
407
415
408
416
## 1.4. Test the output
409
417
410
-
Let's add an assertion to our test to check the output file was created.
411
-
412
-
We can also add it as a separate test, with an informative name.
418
+
Let's add an assertion to our test to check the output file was created. We'll add it as a separate test, with an informative name, to make the results easier to interpret.
413
419
414
420
**Before:**
415
421
@@ -491,7 +497,7 @@ Test Workflow main.nf
491
497
SUCCESS: Executed 2 tests in 15.165s
492
498
```
493
499
494
-
Success! The test passes because the pipeline completed successfully, the correct number of processes ran and the output files were created. This should also show you how useful it is to provide those informative names for your tests.
500
+
Success! The tests pass because the pipeline completed successfully, the correct number of processes ran and the output files were created. This should also show you how useful it is to provide those informative names for your tests.
495
501
496
502
This is just the surface, we can keep writing assertions to check the details of the pipeline, but for now let's move on to testing the internals of the pipeline.
497
503
@@ -507,7 +513,7 @@ Learn how to test a Nextflow process.
507
513
508
514
## 2 Test a Nextflow process
509
515
510
-
We don't have to write tests for every part of the pipeline, but the more tests we have the more comprehensive we can be about the pipeline and the more confident we can be that it's working as expected. In this section we're going to test both processes in the pipeline.
516
+
We don't have to write tests for every part of the pipeline, but the more tests we have the more comprehensive we can be about the pipeline and the more confident we can be that it's working as expected. In this section we're going to test both processes in the pipeline as individual units.
511
517
512
518
### 2.1. Test the `sayHello` process
513
519
@@ -612,8 +618,6 @@ The test fails because the `sayHello` process declares 1 input channel but 0 wer
612
618
**Before:**
613
619
614
620
```groovy title="tests/main.sayhello.nf.test"
615
-
process {
616
-
"""
617
621
test("Should run without failures") {
618
622
619
623
when {
@@ -635,16 +639,12 @@ process {
635
639
}
636
640
637
641
}
638
-
"""
639
-
}
640
642
```
641
643
642
644
**After:**
643
645
644
646
```groovy title="tests/main.sayhello.nf.test"
645
-
process {
646
-
"""
647
-
test("Should run without failures and produce correct output") {
647
+
test("Should run without failures and produce correct output") {
648
648
649
649
when {
650
650
params {
@@ -664,9 +664,6 @@ process {
664
664
}
665
665
666
666
}
667
-
668
-
"""
669
-
}
670
667
```
671
668
672
669
Let's run the test again to see if it works.
@@ -916,9 +913,7 @@ For now, let's re-use the existing data/greetings.csv file using the example we
0 commit comments