FSL 6.0.2's modextrapaths overrides EB's Python with FSL-specific Python.

[crubb]

Hey,

I created FSL-6.0.2-foss-2019a-Python-3.7.2.eb as a copy from the currently available old toolchain 6.0.2 release and ran into problems, since fslpython stuff is included in $PATH and $PYTHONPATH.

Unfortunately, I cannot reproduce the exact problem right now, but essentially I was loading FSL, then using some SciPy/numpy down the line and numpy error'ed out on some multiarray not found problem or similar. If necessary, I can try to reproduce.

I think the following is the root of my problem:

$ module load Python/3.7.2-GCCcore-8.2.0
$ which python3
/software/easybuild4/software/Python/3.7.2-GCCcore-8.2.0/bin/python3
$ module purge
$ module load FSL/6.0.2-foss-2019a-Python-3.7.2
$ which python3
/software/easybuild4/software/FSL/6.0.2-foss-2019a-Python-3.7.2/fsl/fslpython/bin/python3
$ module list 2>&1 | grep Python
 48) Python/3.7.2-GCCcore-8.2.0
 54) VTK/8.2.0-foss-2019a-Python-3.7.2
 65) FSL/6.0.2-foss-2019a-Python-3.7.2

I don’t think it’s a good idea to have that fslpython overrule EB’s Python, which is happening in the modextrapaths:

# Adding the bin from the virtualenv was the only way to get things working...
modextrapaths = {
    'PATH': 'fsl/fslpython/bin',
    'PYTHONPATH': 'fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/',
}

I rebuilt FSL without modextrapaths defined, and double-checked, that the fslpython/envs/fslpython/bin/python3.7 shebang is automatically included in the relevant scripts:

$ head -n1 /software/easybuild4/software/FSL/6.0.2-foss-2019a-Python-2.7.15/fsl/fslpython/envs/fslpython/bin/imcp
#!/software/easybuild4/software/FSL/6.0.2-foss-2019a-Python-2.7.15/fsl/fslpython/envs/fslpython/bin/python3.7

For example imcp works fine:

$ imcp MNI152_T2_1mm test
$ echo $?
0
$ md5sum MNI152_T2_1mm.nii.gz test.nii.gz
b77bc00055bdd7283dba7d038a217b92  MNI152_T2_1mm.nii.gz
b77bc00055bdd7283dba7d038a217b92  test.nii.gz

Therefore, I would suggest dropping modextrapaths from the 6.0.2 easyconfigs - Or is there something I am overlooking?

Thanks!
--crubb

@casparvl tagged by request of @boegel ;)

[casparvl]

@crubb The reason I put it in there initially is that FSL does some evil things during installation. If I remember correctly, their 'intended' installation was to inject a hardcoded shebang in their own Python scripts (i.e. the ones part of FSL) that point to the Python-3.7 that is shipped with their virtualenv. Thus, the devs claimed I did NOT need to load the virtualenv. However, I have never gotten that to work: maybe because the hardcoded shebang in my case got too long, contained special chars, or something like that. Or maybe the devs simply also had a Python-3.7 installed on their system (we don't have a system Python-3). In the end, the only way I got things to work was by actually loading the virtualenv.

That being said, I wrote it for the 2018b toolchain, which has a Python-3.6.3 in there. If you HAPPEN to use a toolchain that is based on the same Python version as the FSL virtualenv (which I believe is Python-3.7), you may indeed get away with not activating the virtualenv and simply running the FSL python scripts with the EasyBuild Python-3.7. Feel free to do so, but If you contribute back, I would comment out my part in the EasyConfig (not remove it) and put a big fat note with it referring to this issue and my original PR. If someone would base a new EasyConfig on yours again, but for a non-Python-3.7 based toolchain he/she might run into my same issues again, so I think it's good if the comment would still be findable in the EasyConfig =)

By the way, if you want to be as sure as possible that your installation works, try running the FSL_feeds test. The devs provide it as a download, I think in more or less the same place as the FSL sources. You'll have to extract the Feeds tarball, and run the RUN script with argument all to run all tests. I'm not exactly sure whether my Python issue showed with imcp, or only with the FEEDS test.

[crubb]

Thanks for pointing out FEEDS. I was not aware of that package, yet.

My system's python3 is Python 3.6, but as you pointed out 2019a and FSL share the Python 3.7 version.

FEEDS runs fine (apart from FSL's MELODIC now depending on libboost_filesystem.so.1.70.0, which I needed to fix by depending on Boost).

@casparvl @boegel - I would suggest commenting out the modextrapaths for FSL 6.0.2 w/ 2019a and leave a warning in the .eb?

FSL Evaluation and Example Data Suite v6.0.2

Warning - FSL version (6.0.2:a4f562d9) not the same as this test
suite version (6.0.2)! Therefore this test may
not run...maybe you should download the latest version of
the test suite and/or FSL.

start time = Mo 13. Jan 11:43:43 UTC 2020
hostname = test
os = Linux test 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


/bin/rm -rf /home/test/feeds/results ; mkdir /home/test/feeds/results


Starting PRELUDE & FUGUE at Mo 13. Jan 11:43:43 UTC 2020
% error = 0.0
% error = 0.0

Starting SUSAN at Mo 13. Jan 11:43:44 UTC 2020
% error = 0.03

Starting SIENAX (including testing BET and FLIRT and FAST) at Mo 13. Jan 11:43:58 UTC 2020
checking error on BET:
% error = 0.0
checking error on FLIRT:
% error = 0.03
checking error on FAST:
checking error on single-image binary segmentation:
% error = 0.37
checking error on partial volume images:
% error = 0.28
% error = 0.41
% error = 0.29
checking error on SIENAX volume outputs:
% error = 0.22
% error = 0.11
% error = 0.51
% error = 0.04
% error = 0.55

Starting BET2 at Mo 13. Jan 11:47:11 UTC 2020
checking error on T1 brain extraction:
% error = 0.0
checking error on skull and scalp surfaces:
% error = 0.13
% error = 0.25
% error = 0.3

Starting FEAT at Mo 13. Jan 11:49:19 UTC 2020
checking error on filtered functional data:
% error = 0.38
checking error on raw Z stat images:
% error = 0.44
% error = 0.29
% error = 0.39
checking error on thresholded Z stat images:
% error = 0.43
% error = 0.4
% error = 0.46
checking error on registration images:
% error = 0.0
% error = 0.01
checking error on position of largest cluster of Talairached zfstat1:
% error = 0.0
% error = 0.0
% error = 0.01
% error = 0.08
% error = 0.17
% error = 0.05

Starting MELODIC at Mo 13. Jan 11:53:34 UTC 2020
% error = 0.66

Starting FIRST at Mo 13. Jan 11:55:37 UTC 2020
% error = 0.68

Starting FDT (bedpost) at Mo 13. Jan 11:57:33 UTC 2020
checking error on bedpost output:
% error = 0.63
% error = 0.01
% error = 0.3
% error = 0.2
% error = 0.07
% error = 0.3
% error = 0.2

Starting FNIRT at Mo 13. Jan 12:03:49 UTC 2020
% error = 0.03

end time = Mo 13. Jan 12:04:15 UTC 2020


All tests passed

real    20m31,948s
user    189m20,718s
sys     0m59,539s

[justbennet]

Would it be worth contacting the FSL people to see if they can provide the 'recipe' for their virtualenv so that what they are doing can be explicitly compared to what you want to do?

[crubb]

At least for me, at the moment, using FSL's build environment and then using the scripts from the virtualenv (like imcp) works fine with the 2019a toolchain. I wouldn't try to replicate the FSL build in a different way.

[mboisson]

FYI, I had to make a virtual environment, which also copied python, in this commit, for FSL 6.0.3 :
ComputeCanada@7b05bcb

I fixed the problem of having a new python in the path by installing the virtual environment separately, then by creating symlinks in FSL's bin folder, carefully avoiding the python, pip, wheel and activate binaries (but symlinking everything else).