Skip to content

Commit 47fa50f

Browse files
sgiavasissgiavasis
authored
Merge pull request #287 from FCP-INDI/dev/templates
🧑‍💻📝 Adding a template or atlas to be packaged in C-PAC
2 parents 58d221a + 3b38324 commit 47fa50f

File tree

5 files changed

+328
-3
lines changed

5 files changed

+328
-3
lines changed

‎docs/_sources/developer/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ Contents:
1717

1818
installation
1919
pipeline
20+
templates
2021
nodes
2122
random_state
2223
logging

‎docs/_sources/developer/templates.rst‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
Templates and Atlases
2+
=====================
3+
4+
Templates and altases included in the C-PAC Docker and Singularity images are maintained in a separate repository: `FCP-INDI/C-PAC_templates <https://github.com/FCP-INDI/C-PAC_templates>`_ and packaged in a |staging image|_.
5+
6+
Adding a template or atlas to be packaged in C-PAC
7+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
8+
9+
In a branch:
10+
11+
#. For the new file(s), fill in as much of the appropriate table(s) in |README.md|_ as you can. If no appropriate table exists in the ``README``, add one.
12+
#. If you had add a new table to the ``README``, add a matching subdirectory under |atlases/label|_.
13+
#. Rename your file(s) to begin with the "Atlas name" you used in the ``README`` table, and use BIDS standard template identifiers :cite:`cite-dev-templates-BIDS22VIIItemplateId` to identify the atlas/template's space if possible.
14+
#. Add your file(s).
15+
16+
* If your file(s) are available from a reliable source,
17+
18+
#. If they're available in a reliable, web-accessible Docker image, `use that image as a staging image <https://docs.docker.com/develop/develop-images/multistage-build/>`_ and ``COPY`` the files directly from , e.g.
19+
20+
.. code:: Dockerfile
21+
22+
COPY --from=ghcr.io/fcp-indi/c-pac/neuroparc:v1.0-human /ndmg_atlases /ndmg_atlases
23+
24+
#. If they're avilable in a GitHub repository,
25+
26+
#. Clone into the ``GitHubPipelines`` staging image, `e.g. <https://github.com/FCP-INDI/C-PAC_templates/blob/9b33a4f1/.github/Dockerfiles/cpac_templates.Dockerfile#L6-L19>`_,
27+
28+
.. literalinclude:: /references/cpac_templates.Dockerfile
29+
:language: Dockerfile
30+
:lines: 6,19
31+
32+
#. Copy or move relevant files to a directory that does not contain any unnecessary files, `e.g. <https://github.com/FCP-INDI/C-PAC_templates/blob/9b33a4f1/.github/Dockerfiles/cpac_templates.Dockerfile#L7-L25>`_
33+
34+
.. literalinclude:: /references/cpac_templates.Dockerfile
35+
:language: Dockerfile
36+
:lines: 7-18,20-25
37+
38+
* Otherwise, your file(s) to the appropriate subdirectory of |atlases/label|_ in your branch.
39+
#. If adding to a directory not already being copied from the ``GitHubPipelines`` staging image to the final templates image, add a ``COPY`` command to copy the files over, `e.g. <https://github.com/FCP-INDI/C-PAC_templates/blob/9b33a4f1/.github/Dockerfiles/cpac_templates.Dockerfile#L39-L40>`_,
40+
41+
.. literalinclude:: /references/cpac_templates.Dockerfile
42+
:language: Dockerfile
43+
:lines: 39-40
44+
45+
If your template or atlas is in a different path
46+
------------------------------------------------
47+
If you're adding your template or atlas to the |staging image|_ in a location other than ``/cpac_templates/``, update both the `Bionic Beaver staging image <https://github.com/FCP-INDI/C-PAC/blob/6ab438f7/.github/Dockerfiles/Ubuntu.bionic-non-free.Dockerfile#L153-L156>`_ and the `Xenial Xerus staging image <https://github.com/FCP-INDI/C-PAC/blob/6ab438f752de1de764378ffc73a423cfa36c4b44/.github/Dockerfiles/Ubuntu.xenial-20200114.Dockerfile#L163-L166>`_ to ``COPY`` your files from the |staging image|_ to each Ubuntu staging image, e.g.,
48+
49+
.. literalinclude:: /references/Ubuntu.bionic-non-free.Dockerfile
50+
:language: Dockerfile
51+
:lines: 153-156
52+
53+
54+
Testing your updates
55+
--------------------
56+
57+
You can use `.github/Dockerfiles/cpac_templates.Dockerfile <https://github.com/FCP-INDI/C-PAC_templates/blob/main/.github/Dockerfiles/cpac_templates.Dockerfile>`_ to build a custom |staging image|_, e.g.,
58+
59+
.. code:: BASH
60+
61+
docker build -t cpac_templates:custom -f .github/Dockerfiles/cpac_templates.Dockerfile . # <-- run in your modified fork or branch of FCP-INDI/C-PAC_templates
62+
docker run --rm -dit \
63+
--name=CPACTemplates \
64+
-v cpac_templates:/cpac_templates \
65+
ghcr.io/fcp-indi/c-pac_templates:latest # <-- Start a daemon image with your custom templates
66+
docker run --rm -it \
67+
--volumes-from CPACTemplates \
68+
fcpindi/c-pac:nightly $BIDS_DIR $OUTPUTS_DIR participant # <-- Use C-PAC as usual, but bind the custom /cpac_templates
69+
docker attach CPACTemplates # <-- Clean up when you're done
70+
exit # <-- Clean up when you're done
71+
72+
73+
.. note::
74+
75+
When you use ``--volumes-from`` to bind a directory that already exists in the image, the container will only be able to see the content from the bound daemon container, not any of the files in the original image. You can bind specific files instead of their parent directories to make files from the bound daemon container available in an existing directory in a container.
76+
77+
Once your changes have been merged to |main|_, C-PAC will include them in each subsequent build.
78+
79+
References
80+
^^^^^^^^^^
81+
82+
.. bibliography::
83+
:cited:
84+
:keyprefix: cite-dev-templates-
85+
86+
.. hyperlink formatting
87+
88+
.. |atlases/label| replace:: ``atlases/label``
89+
90+
.. _atlases/label: https://github.com/FCP-INDI/C-PAC_templates/tree/main/atlases/label
91+
92+
.. |main| replace:: ``main``
93+
94+
.. _main: https://github.com/FCP-INDI/C-PAC_templates/tree/main
95+
96+
.. |README.md| replace:: ``README.md``
97+
98+
.. _README.md: https://github.com/FCP-INDI/C-PAC_templates/blob/main/README.md
99+
100+
.. |staging image| replace:: ``c-pac_templates`` staging image
101+
102+
.. _staging image: https://github.com/FCP-INDI/C-PAC_templates/pkgs/container/c-pac_templates

‎docs/_sources/references/Ubuntu.bionic-non-free.Dockerfile‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
FROM neurodebian:bionic-non-free AS dcan-hcp
2+
FROM ghcr.io/fcp-indi/c-pac_templates:latest as c-pac_templates
3+
4+
ARG DEBIAN_FRONTEND=noninteractive
5+
6+
# Adding DCAN dependencies & HCP code
7+
RUN apt-get update && \
8+
apt-get install -y git && \
9+
mkdir -p /opt/dcan-tools && \
10+
git clone -b 'v2.0.0' --single-branch --depth 1 https://github.com/DCAN-Labs/DCAN-HCP.git /opt/dcan-tools/pipeline
11+
12+
# using neurodebian runtime as parent image
13+
FROM neurodebian:bionic-non-free
14+
LABEL org.opencontainers.image.description "NOT INTENDED FOR USE OTHER THAN AS A STAGE IMAGE IN A MULTI-STAGE BUILD \
15+
Ubuntu Bionic base image"
16+
ARG DEBIAN_FRONTEND=noninteractive
17+
18+
ENV TZ=America/New_York
19+
20+
# Creating usergroup and user
21+
# Allow users to update / create themselves
22+
# Installing system requirments, the BIDS validator & minconda
23+
RUN groupadd -r c-pac && \
24+
useradd -r -g c-pac c-pac_user && \
25+
mkdir -p /home/c-pac_user/ && \
26+
chown -R c-pac_user:c-pac /home/c-pac_user && \
27+
chmod 777 / && \
28+
chmod ugo+w /etc/passwd && \
29+
apt-get update && \
30+
apt-get install -y --no-install-recommends \
31+
apt-utils \
32+
apt-transport-https \
33+
bc \
34+
build-essential \
35+
bzip2 \
36+
ca-certificates \
37+
cmake \
38+
curl \
39+
dh-autoreconf \
40+
git \
41+
gnupg \
42+
graphviz \
43+
graphviz-dev \
44+
gsl-bin \
45+
libcanberra-gtk-module \
46+
libexpat1-dev \
47+
libgiftiio-dev \
48+
libglib2.0-dev \
49+
libglu1-mesa \
50+
libglu1-mesa-dev \
51+
libjpeg-progs \
52+
libgl1-mesa-dri \
53+
libglw1-mesa \
54+
libgsl-dev \
55+
libmotif-dev \
56+
libtool \
57+
libx11-dev \
58+
libxext-dev \
59+
libxft2 \
60+
libxft-dev \
61+
libxi-dev \
62+
libxml2 \
63+
libxml2-dev \
64+
libxmu-dev \
65+
libxmu-headers \
66+
libxpm-dev \
67+
libxslt1-dev \
68+
locales \
69+
m4 \
70+
make \
71+
mesa-common-dev \
72+
mesa-utils \
73+
netpbm \
74+
ninja-build \
75+
openssh-client \
76+
pkg-config \
77+
rsync \
78+
software-properties-common \
79+
tcsh \
80+
unzip \
81+
vim \
82+
wget \
83+
x11proto-xext-dev \
84+
x11proto-print-dev \
85+
xauth \
86+
xutils-dev \
87+
xvfb \
88+
zlib1g-dev && \
89+
export XDG_CONFIG_HOME=/usr/bin && \
90+
curl https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.2/install.sh | bash && \
91+
export NVM_DIR=/usr/bin/nvm && \
92+
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" && \
93+
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" && \
94+
nvm install 12.12.0 && \
95+
nvm use 12.12.0 && \
96+
nvm alias default 12.12.0 && \
97+
npm install --global npm@^7 && \
98+
npm install -g bids-validator && \
99+
curl -sLo /tmp/libpng12.deb http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb && \
100+
dpkg -i /tmp/libpng12.deb && \
101+
rm /tmp/libpng12.deb && \
102+
add-apt-repository --yes ppa:zeehio/libxp && \
103+
apt-get update && \
104+
apt-get install --no-install-recommends \
105+
libxp6 \
106+
libxp-dev && \
107+
add-apt-repository --remove --yes ppa:zeehio/libxp && \
108+
apt-get update && \
109+
curl -sO https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh && \
110+
bash Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -p /usr/local/miniconda && \
111+
rm Miniconda3-py37_4.8.2-Linux-x86_64.sh && chmod -R 777 /usr/local/miniconda && \
112+
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
113+
echo $TZ > /etc/timezone && \
114+
sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
115+
dpkg-reconfigure --frontend=noninteractive locales && \
116+
update-locale LANG="en_US.UTF-8" && \
117+
chmod 777 /opt && \
118+
chmod a+s /opt
119+
120+
ENV PATH=/usr/bin/nvm/versions/node/v12.12.0/bin:/usr/local/miniconda/bin:$PATH
121+
122+
# Installing conda dependencies, torch & Python dependencies
123+
COPY requirements.txt /opt/requirements.txt
124+
RUN conda update conda -y && \
125+
conda install nomkl && \
126+
conda install -y \
127+
blas \
128+
cython \
129+
matplotlib==3.1.3 \
130+
networkx==2.4 \
131+
nose==1.3.7 \
132+
numpy==1.16.4 \
133+
pandas==0.23.4 \
134+
scipy==1.4.1 \
135+
traits==4.6.0 \
136+
wxpython \
137+
pip && \
138+
pip install \
139+
torch==1.2.0 torchvision==0.4.0 -f https://download.pytorch.org/whl/torch_stable.html && \
140+
pip install --upgrade setuptools && \
141+
pip install --upgrade pip && \
142+
pip install -r /opt/requirements.txt && \
143+
pip install xvfbwrapper && \
144+
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
145+
apt-get install -y --no-install-recommends git-lfs && \
146+
git lfs install
147+
148+
# Installing C-PAC templates and atlases
149+
COPY --from=c-pac_templates /cpac_templates /cpac_templates
150+
COPY --from=dcan-hcp /opt/dcan-tools/pipeline/global /opt/dcan-tools/pipeline/global
151+
COPY --from=ghcr.io/fcp-indi/c-pac/neuroparc:v1.0-human /ndmg_atlases /ndmg_atlases
152+
153+
# Installing surface files for downsampling
154+
COPY --from=c-pac_templates /opt/dcan-tools/pipeline/global/templates/standard_mesh_atlases/ /opt/dcan-tools/pipeline/global/templates/standard_mesh_atlases/
155+
COPY --from=c-pac_templates /opt/dcan-tools/pipeline/global/templates/28224_Greyordinates/ /opt/dcan-tools/pipeline/global/templates/28224_Greyordinates/
156+
COPY --from=c-pac_templates /opt/dcan-tools/pipeline/global/templates/8617_Greyordinates/ /opt/dcan-tools/pipeline/global/templates/8617_Greyordinates/
157+
158+
ENTRYPOINT ["/bin/bash"]
159+
160+
# Link libraries for Singularity images
161+
RUN ldconfig && \
162+
apt-get clean && \
163+
apt-get autoremove -y && \
164+
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
165+
166+
# set user
167+
USER c-pac_user

‎docs/_sources/references/cpac_templates.Dockerfile‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Get templates from other GitHub repositories
2+
FROM alpine/git AS GitHubPipelines
3+
4+
ARG DEBIAN_FRONTEND=noninteractive
5+
6+
RUN git clone --branch v0.1.1 --depth 1 https://github.com/DCAN-Labs/dcan-macaque-pipeline.git && \
7+
# Washington University NHP Pipelines
8+
cd dcan-macaque-pipeline/global/templates && \
9+
mkdir -p /macaque_templates && \
10+
for FILE in $(ls MacaqueYerkes19*mm.nii.gz MacaqueYerkes19*mm_brain.nii.gz MacaqueYerkes19*brain_mask.nii.gz); \
11+
do cp $FILE /macaque_templates/$FILE; \
12+
done && \
13+
for FILE in $(ls JointLabelCouncil/MacaqueYerkes19_T1w_0.5mm/); \
14+
do cp JointLabelCouncil/MacaqueYerkes19_T1w_0.5mm/$FILE /macaque_templates/MacaqueYerkes19_T1w_0.5mm_desc-JLC_$FILE; \
15+
done && \
16+
for FILE in $(ls JointLabelCouncil/J_Macaque_11mo_atlas_nACQ_194x252x160space_0.5mm/); \
17+
do cp JointLabelCouncil/J_Macaque_11mo_atlas_nACQ_194x252x160space_0.5mm/$FILE /macaque_templates/J_Macaque_11mo_atlas_nACQ_194x252x160space_0.5mm_desc-JLC_$FILE; \
18+
done && \
19+
git clone --branch v0.22.6-Edit_KRR_LITE_to_only_allow_for_corr_kernel https://github.com/ThomasYeoLab/CBIG.git && \
20+
# CBIG
21+
mkdir -p /atlases/label/human && \
22+
for NODECOUNT in 200 300 400 1000; \
23+
do \
24+
cp CBIG/stable_projects/brain_parcellation/Schaefer2018_LocalGlobal/Parcellations/MNI/Schaefer2018_${NODECOUNT}Parcels_17Networks_order_FSLMNI152_2mm.nii.gz /atlases/label/human/Schaefer2018_space-FSLMNI152_res-2mm_desc-${NODECOUNT}Parcels17NetworksOrder.nii.gz; \
25+
done
26+
27+
# using neurodebian runtime as parent image
28+
FROM neurodebian:bionic-non-free
29+
30+
ARG DEBIAN_FRONTEND=noninteractive
31+
32+
# create usergroup and user
33+
RUN groupadd -r c-pac && \
34+
useradd -r -g c-pac c-pac_user && \
35+
mkdir -p /home/c-pac_user/ && \
36+
chown -R c-pac_user:c-pac /home/c-pac_user
37+
38+
# Move all templates into /cpac_templates
39+
COPY --from=GitHubPipelines /macaque_templates/* /cpac_templates/
40+
COPY --from=GitHubPipelines /atlases/label/human/* /cpac_templates/
41+
ADD atlases/label/*/* /cpac_templates/
42+
43+
# set user
44+
USER c-pac_user

‎docs/_sources/references/references.bib‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,24 @@ @article{Abra14
1010
doi = {10.3389/fninf.2014.00014}
1111
}
1212

13+
@incollection{BIDS22VIIItemplateId,
14+
title = {Appendix {{VIII}}: {{Coordinate}} systems \S{} Standard template identifiers},
15+
booktitle = {The {{Brain Imaging Data Structure}} ({{BIDS}}) {{Specification}}},
16+
author = {{BIDS-contributors}},
17+
year = {2022},
18+
edition = {1.7.0},
19+
url = {https://bids-specification.readthedocs.io/en/v1.7.0/99-appendices/08-coordinate-systems.html#standard-template-identifiers},
20+
copyright = {Copyright \textcopyright{} 2018-2022, BIDS Contributors - CC BY 4.0}
21+
}
22+
1323
@incollection{BIDS21,
1424
title = {Appendix {{IX}}: {{Entities}}},
1525
booktitle = {The {{Brain Imaging Data Structure}} ({{BIDS}}) {{Specification}}},
1626
author = {{BIDS-contributors}},
17-
year = {2021},
18-
edition = {1.6.0},
19-
url = {https://bids-specification.readthedocs.io/en/v1.6.0/99-appendices/09-entities.html}
27+
year = {2022},
28+
edition = {1.7.0},
29+
url = {https://bids-specification.readthedocs.io/en/v1.7.0/99-appendices/09-entities.html},
30+
copyright = {Copyright \textcopyright{} 2018-2022, BIDS Contributors - CC BY 4.0}
2031
}
2132

2233
@incollection{Ciri19,

0 commit comments

Comments
 (0)