Skip to content

Commit af92dbd

Browse files
Nauman702Nauman702
authored
Merge pull request #7 from nipraxis-fall-2022-forks/analysis-sketch
Sketch of analysis plan
2 parents fe819c4 + 85c7b9c commit af92dbd

File tree

3 files changed

+275
-1
lines changed

3 files changed

+275
-1
lines changed

‎analysis_plan.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Analysis plan
2+
3+
This is file for your sketch for an *analysis plan*.
4+
5+
See the instructions in `on_the_project.md`.
6+
7+
Feel free to edit anything here. You may well want to digitally burn the
8+
instructions here after reading, to make space for your own thoughts.

‎findoutlie/outfind.py‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,33 @@
33

44
from pathlib import Path
55

6+
import numpy as np
7+
8+
import nibabel as nib
9+
10+
from .metrics import dvars
11+
from .detectors import iqr_detector
12+
613

714
def detect_outliers(fname):
8-
return [42]
15+
""" Detect outliers given image file path `filename`
16+
17+
Parameters
18+
----------
19+
fname : str or Path
20+
Filename of 4D image, as string or Path object
21+
22+
Returns
23+
-------
24+
outliers : array
25+
Indices of outlier volumes.
26+
"""
27+
# This is a very simple function, using dvars and iqroutliers
28+
img = nib.load(fname)
29+
dvs = dvars(img)
30+
is_outlier = iqr_detector(dvs, iqr_proportion=2)
31+
# Return indices of True values from Boolean array.
32+
return np.nonzero(is_outlier)
933

1034

1135
def find_outliers(data_directory):

‎on_the_project.md‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
# On the diagnostics project
2+
3+
These instructions introduce your task for the next few weeks, working on the
4+
project. Specifically, these instructions are about the pull request (PR) that
5+
contain these instructions, and how to get going on your analysis plan.
6+
7+
We should say to start off, that the term *analysis plan* is a bit grand. It
8+
should better be called an analysis sketch.
9+
10+
The purpose of this PR is:
11+
12+
1. Practice on some Git / Github collaboration *with us and with each other*.
13+
2. Practice on editing [Markdown text](https://www.markdowntutorial.com).
14+
3. Giving you a chance to ask us questions about the project.
15+
4. Making sure you're ready to get going with improving code to detect
16+
outliers.
17+
5. Giving you more information on your task.
18+
6. Making a sketch of what you want to do over the next week or two for the
19+
project.
20+
21+
## Github practice, questions
22+
23+
You are going to get this file, and several others, as a *pull request* (PR) to
24+
your repository.
25+
26+
Your first job is to use this PR to ask questions of us, your instructors.
27+
28+
What we propose you do, is use the PR interface to ask for clarification about
29+
the task, or the project. You can enter comments in the PR interface, or go
30+
the "Files changed" tabs, and click on individual lines to add comments or
31+
questions about specific lines in the file.
32+
33+
Use the tag `@nipraxis-fall-2022/instructors` to point us to your questions.
34+
35+
Once you are happy you've understood the task, merge this PR.
36+
37+
## On Markdown
38+
39+
The file is in Markdown format, and you will be writing an analysis plan, also
40+
in Markdown.
41+
42+
Markdown is a *markup language*. A Markdown file is a conventional text file
43+
that you can open in any text editor. The special aspect of a Markdown file is
44+
the *markup*. Markup consists of special bits of text that specify
45+
*formatting* of the text. For example, in order to make a word in **bold**
46+
text, using Markdown markup, you put two asterisks either side of the text you
47+
want to be in bold. When you want a properly formatted version of your
48+
Markdown file, you convert it to the formatted version, using a *Markdown
49+
renderer*. A Markdown renderer is some system that can interpret the Markdown
50+
markup and display the text as you intended, with bold text as bold, headings
51+
as headings and so on.
52+
53+
There are very many Markdown renderers, but the Github site is one. When you
54+
put a `.md` file into your repository, like this one, and then navigate to the
55+
relevant file in the Github web interface, you will see that Github has
56+
*rendered* the Markdown formatting, showing bold as **bold**, headings as
57+
headings, and so on.
58+
59+
Markdown has become the standard way of writing text files with markup, and you
60+
will see it everywhere on systems that programmers use, such as Github, and in
61+
the Jupyter notebook.
62+
63+
Markdown has many dialects, meaning that there is some markup that every
64+
Markdown renderer understands, such as **bold**, and other markup that only
65+
some renderers understand. The Markdown that every renderer understands is
66+
called [standard Markdown](https://www.markdownguide.org/basic-syntax). Github
67+
has its own dialect of Markdown, called [Github flavored
68+
Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github).
69+
You can usually stick to the standard stuff, but you may need to consult the
70+
Github documents if you want to do something slightly more fancy, like a table.
71+
72+
## Making sure you're ready
73+
74+
To be ready to get going on your project you need to make sure you have merged these three PRs:
75+
76+
* "add-dvars"
77+
* "Add machinery to install module directory."
78+
* "Fix use of Path in find_outliers script"
79+
80+
Make sure you've done the exercises there. Run the following checks, from the
81+
homeworks:
82+
83+
```
84+
# You should see no errors.
85+
python3 scripts/validate_data.py
86+
```
87+
88+
```
89+
# You should see: "Tests passed".
90+
python3 findoutlie/tests/test_detectors.py
91+
```
92+
93+
```
94+
# You should see "=== ? passed in ? seconds ==="
95+
# Where ? are numbers that will depend on your system and repository.
96+
pytest findoutlie
97+
```
98+
99+
If you don't get these outputs, check back with us by tagging use with a
100+
question on this PR.
101+
102+
Next, have a look at the `findoutlie/outfind.py` *in this PR*. You will see a
103+
basic implementation of outlier detection using your DVARs implementation, from
104+
the homework.
105+
106+
*After you have merged this PR*, you can run:
107+
108+
```
109+
python3 scripts/find_outliers.py data
110+
```
111+
112+
and you should see the default DVARS detection of outliers, giving something
113+
like this (the exact output will depend on your own data):
114+
115+
```
116+
data/group-00/sub-08/func/sub-08_task-taskzero_run-01_bold.nii.gz, [129 133 134]
117+
data/group-00/sub-08/func/sub-08_task-taskzero_run-02_bold.nii.gz, [2]
118+
data/group-00/sub-01/func/sub-01_task-taskzero_run-01_bold.nii.gz, []
119+
...
120+
ata/group-00/sub-03/func/sub-03_task-taskzero_run-01_bold.nii.gz, [ 0 25 26 75 77 78 79 80 102 103 129 156 160]
121+
data/group-00/sub-04/func/sub-04_task-taskzero_run-01_bold.nii.gz, []
122+
data/group-00/sub-04/func/sub-04_task-taskzero_run-02_bold.nii.gz, [ 22 23 76 77 103 104]
123+
```
124+
125+
## Your task
126+
127+
This is already an outlier detection method, but a very crude one, using a
128+
fixed threshold of 2 x the interquartile range on the DVARS values to detect
129+
outliers.
130+
131+
Your job, should you chose to accept it, is to improve the code so that the
132+
`find_outliers.py` script does a better job at detecting outliers.
133+
134+
How do you know you have done a good job? Well - that is the key question.
135+
136+
At a first pass, we would like you to *investigate* the FMRI time-series, by
137+
looking at various measures of the scans, and looking at the scans themselves,
138+
to see whether you can identify artifacts.
139+
140+
In due course, the thing we are going to evaluate, is how well you *recover the
141+
activation pattern*, when you exclude these scans. By *recover the activation
142+
pattern*, we mean, how well does a statistical analysis do, using the task
143+
regressors, in finding the activation pattern, after you exclude the outliers?
144+
In particular, do you do a better job of recovering the activation pattern
145+
after removing the outliers? And can removing another set of outliers do a
146+
better job?
147+
148+
But how will you tell whether you are doing a better job of recovering the
149+
activation?
150+
151+
We will soon send you another PR, that gives you a basic script to do a
152+
statistical analysis on an FMRI run, and generate an activation image, given
153+
some identified outlier scans. This will use the machinery we will be teaching
154+
over the next few weeks. But even this is not automated. So, part of your job
155+
here is to look at the activation images to see if you believe the result,
156+
after your outlier estimation.
157+
158+
We will do something more sophisticated, and you may want to replicate this later.
159+
We will use other datasets (that you don't have) from the same FMRI series, to
160+
estimate the correct activation, and then compare your activation estimate,
161+
after excluding outliers, to the estimate from other datasets. If you've done
162+
a good job, your estimate will be closer to the estimation from the other
163+
datasets, on the assumption that the datasets do not share noise from their
164+
outliers. We will talk more about this in later sessions. But, for now, your
165+
job will be to look at how you are doing, by eye.
166+
167+
You should add a text file giving a brief summary for each outlier scan, why
168+
you think the detected scans should be rejected as an outlier, and your
169+
educated guess as to the cause of the difference between this scan and the rest
170+
of the scans in the run.
171+
172+
## Grading
173+
174+
We will rate you on:
175+
176+
* The quality of your outlier detection as assessed by the improvement in the
177+
statistical testing for the experimental model after removing the outliers — as
178+
above.
179+
* The generality of your outlier detection as assessed by the improvement in
180+
the statistical testing for the experimental model after removing the
181+
outliers, for another similar dataset.
182+
* The quality of your code. How easy is your code to read, and understand? Is
183+
it well formatted, and well organized into different files and functions.
184+
* The quality and transparency of your process, from your interactions on
185+
Github.
186+
* The quality of your arguments about the scans rejected as outliers, in the
187+
text file above.
188+
189+
Your outlier detection script should be *reproducible*.
190+
191+
That means that we, your instructors, should be able to clone your repository,
192+
and then follow simple instructions in order to be able to reproduce your run
193+
of `scripts/find_outliers.py data` and get the same answer.
194+
195+
To make this possible, fill out the `README.md` text file in your repository to
196+
describe a few simple steps that we can take to set up on our own machines and
197+
run your code. Have a look at the current `README.md` file for a skeleton. We
198+
should be able to perform these same steps to get the same output as you from
199+
the outlier detection script.
200+
201+
## The sketch
202+
203+
The purpose of the `analysis_plan.md` document is for you to record your first
204+
thoughts about how you will approach the problem.
205+
206+
* Do you need to arrange times to meet online or IRL to discuss progress, or
207+
can you collaborate by messaging back and forth via the Github interface, PRs
208+
and issues?
209+
* What will you explore for your outlier detection? For example, the current
210+
script only uses DVARS with a fixed threshold — will you use other metrics
211+
instead, or as well? What metrics? Will you want to adjust thresholds by
212+
hand? Or work out automatic thresholds? What would the interface to such
213+
code look like?
214+
* Do you want to consider more advanced techniques such as [Principal Component
215+
Analysis](https://matthew-brett.github.io/teaching/pca_introduction.html) or
216+
even [Independent Component
217+
Analysis](https://en.wikipedia.org/wiki/Independent_component_analysis)? We
218+
won't cover those techniques in this course, so if you use them, you should
219+
make sure you explain them in your write-up, and show us that you understand
220+
them to a reasonable level.
221+
* Even for DVARS - how will you use the values? For example, imagine someone
222+
moves instantaneously between scans 5 and 6. There is a big DVARS value
223+
between 5 and 6 because of the movement signal in 6, so 6 may be an outlier —
224+
but what would you expect to see in scan 7? If scan 7 is pretty similar to
225+
scan 6, it is also an outlier?
226+
* You do not have to restrict yourself to just identifying outliers if you
227+
would prefer to go further. For example, you could also propose regressors
228+
to go into your statistical estimation to allow for any artifacts that you
229+
have detected. If so, you will need to create these regressors, and explain
230+
how they should be used, giving reproducible code for their use on your given
231+
dataset.
232+
* We suggest you plan a literature review on outlier detection in functional
233+
MRI, and write this into your plan, and summarize in your project files in due
234+
course.
235+
236+
## That's it
237+
238+
Good luck.
239+
240+
Remember to ask for help early and often.
241+
242+
Now on to `analysis_plan.md`.

0 commit comments

Comments
 (0)