ENH proposal: use neurodocker to generate and minimize containers for testing

[satra]

@oesteban @effigies @chrisfilo @djarecka

i would like to change nipype's container generation and testing using the following approach:

1. generate dockerfiles using neurodocker
2. check if dockerfile hash has changed or force update is set (need to determine this from PR)
3. if force update, regenerate dockerfile
4. build docker image if needed or reload minimized image
5. run all tests via neurodocker reprozip (in parallel)
6. combine all reprozip packs into a single pack
7. generate minimized docker image using reprounzip and cache this image

if this sounds good, perhaps @kaczmarj can work on implementing this.

[chrisgorgo]

I wonder if it would make sense to push the final image to a dedicated docker hub repo.

…

On Sat, Aug 5, 2017 at 9:29 AM, Satrajit Ghosh ***@***.***> wrote: @oesteban <https://github.com/oesteban> @effigies <https://github.com/effigies> @chrisfilo <https://github.com/chrisfilo> @djarecka <https://github.com/djarecka> i would like to change nipype's container generation and testing using the following approach: 1. generate dockerfiles using neurodocker 2. check if dockerfile hash has changed or force update is set (need to determine this from PR) 3. if force update, regenerate dockerfile 4. build docker image if needed or reload minimal dockerfile 5. run all tests via neurodocker reprozip (in parallel) 6. combine all reprozip packs into a single pack 7. generate minimized docker image using reprounzip and cache this image if this sounds good, perhaps @kaczmarj <https://github.com/kaczmarj> can work on implementing this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2144>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOkp3qLFhMzqxTUGzrsPrNFZpQxLiSaks5sVJhsgaJpZM4OugFu> .

[satra]

@chrisfilo - yes - we should push the minimized image to a docker hub repo with a version tag.

[kaczmarj]

here are neurodocker commands to generate the base and main dockerfiles.

base

docker run --rm kaczmarj/neurodocker:latest generate \
--base neurodebian:xenial-non-free --pkg-manager apt \
--freesurfer version=6.0.0 min=true \
--install afni ants apt-utils bzip2 file fsl-core fsl-mni152-templates \
          fusefat g++ git graphviz make ruby unzip xvfb \
--add-to-entrypoint "source /etc/fsl/fsl.sh" "source /etc/afni/afni.sh" \
--env ANTSPATH=/usr/lib/ants PATH='/usr/lib/ants:$PATH'  \
--c3d version=1.0.0 \
--instruction "RUN gem install fakes3" \
--spm version=12 matlab_version=R2017a \
--workdir /work

main

docker run --rm kaczmarj/neurodocker:latest generate \
--base nipype/base --pkg-manager apt \
--copy docker/files/run_* nipype/external/fsl_imglob.py /usr/bin/ \
--copy . /src/nipype \
--env MKL_NUM_THREADS=1 OMP_NUM_THREADS=1 \
--miniconda env_name=neuro \
            python_version=3.6 \
            conda_install="icu=58.1 libxml2 libxslt matplotlib mkl numpy pandas
                           psutil scikit-learn scipy traits=4.6.0" \
            pip_opts="-e" \
            pip_install="/src/nipype[all]"

[kaczmarj]

@satra - can you please clarify number 2?

2. check if dockerfile hash has changed or force update is set (need to determine this from PR)

checking whether the base dockerfile changed is straightforward, but the nipype source is copied into the main docker image, so i would be inclined to rebuild the main docker image with each PR. or is this what you meant by "need to determine this from PR"?

[satra]

@kaczmarj - step 2 is more about base than main. yes main should be rebuilt with every code change.