Skip to content

Commit 8af6a37

Browse files
jwokatyjwokaty
authored
Separate installation for gpu/nongpu (#160)
* Separate installation for gpu/nongpu * Add gpu pillar * Add default name * Fix equality * Clone repo * Revert "Clone repo" This reverts commit 14902cd. * Add gpu pillars * Remove warning * Make build_user * Add required to compile apt pkgs * Fix artifact names * Add Nvidia notice, update versions in container section
1 parent 012ad6c commit 8af6a37

File tree

12 files changed

+121
-32
lines changed

12 files changed

+121
-32
lines changed

‎.github/workflows/build_ghcr.yaml‎

Lines changed: 18 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ jobs:
6969
build-args: |
7070
CYCLE=${{matrix.branch}}
7171
BASE_IMAGE=${{matrix.base.image}}
72+
NAME=${{matrix.base.name}}
7273
labels: ${{ steps.meta.outputs.labels }}
7374
outputs: type=image,name=${{ steps.vars.outputs.container }},push-by-digest=true,name-canonical=true,push=true
7475
platforms: ${{matrix.platform}}
@@ -151,22 +152,30 @@ jobs:
151152
strategy:
152153
fail-fast: false
153154
matrix:
154-
platform: [linux/amd64]
155-
branch: [devel, release]
156-
base:
157-
- {image: "ubuntu:noble", name: "noble"}
158-
- {image: "nvcr.io/nvidia/cuda:12.9.0-devel-ubuntu24.04", name: "nvidia-noble"}
155+
ubuntu: [noble]
159156
steps:
160-
- name: Download digests devel
157+
- name: Download digests devel nobel
158+
uses: actions/download-artifact@v4
159+
with:
160+
name: digests-devel-noble-${{ github.run_id }}
161+
path: /tmp/digests-devel
162+
163+
- name: Download digests devel nvidia-nobel
161164
uses: actions/download-artifact@v4
162165
with:
163-
name: digests-devel-${{ matrix.base.name }}-${{ github.run_id }}
166+
name: digests-devel-nvidia-noble-${{ github.run_id }}
164167
path: /tmp/digests-devel
165168

166-
- name: Download digests release
169+
- name: Download digests release noble
170+
uses: actions/download-artifact@v4
171+
with:
172+
name: digests-release-noble-${{ github.run_id }}
173+
path: /tmp/digests-release
174+
175+
- name: Download digests release nvidia-noble
167176
uses: actions/download-artifact@v4
168177
with:
169-
name: digests-release-${{ matrix.base.name }}-${{ github.run_id }}
178+
name: digests-release-nvidia-noble-${{ github.run_id }}
170179
path: /tmp/digests-release
171180

172181
- run: mkdir -p /tmp/digests && mv /tmp/digests-devel/* /tmp/digests/ && mv /tmp/digests-release/* /tmp/digests/

‎Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
ARG BASE_IMAGE=ubuntu:noble
22
FROM ${BASE_IMAGE} AS build
33
ARG CYCLE=release
4+
ARG NAME=
45
RUN useradd -ms /bin/bash biocbuild && apt update -qq && apt install sudo systemd -y && usermod -aG sudo biocbuild && echo "biocbuild ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
56
USER biocbuild
67
COPY . /home/biocbuild/bioconductor_salt
78
WORKDIR /home/biocbuild
8-
RUN DEBIAN_FRONTEND="noninteractive" bash bioconductor_salt/startup_bbs_standalone_${CYCLE}.sh
9+
RUN DEBIAN_FRONTEND="noninteractive" bash bioconductor_salt/startup_bbs_standalone_${CYCLE}.sh ${NAME}
910

1011
FROM ${BASE_IMAGE} AS final
1112
COPY --from=build / /

‎README.md‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,29 +7,46 @@ It also builds [BBS-like
77
containers](https://github.com/Bioconductor/bioconductor_salt/pkgs/container/bioconductor_salt).
88

99
### Simulating the BBS Ubuntu environment in a container
10-
We are experimentally building and publishing containers under the name `ghcr.io/bioconductor/bioconductor_salt`,
10+
11+
We are building and publishing containers under the name `ghcr.io/bioconductor/bioconductor_salt`,
1112
which can be used to mimic a BBS-like linux environment, in hopes of easing reproducibility and interactive debugging
1213
of the BBS environment for package developers.
13-
We currently offer containers for both `release` and `devel` Bioconductor versions with Ubuntu `jammy` (`22.04`).
14+
15+
We currently offer containers for both `release` and `devel` Bioconductor versions with Ubuntu `noble` (`24.04`).
1416
Container tags with various version pinnings can be used to acquire a particular environment, following the schema
15-
`[ubuntu_version]-bioc-[bioc_version]-r-[r_version]` eg `jammy-bioc-3.18-r-4.3.2` or `22.04-bioc-3.18-r-4.3.2`, where
16-
each level is optional. For example, one could use tag `jammy-bioc-3.18` or `22.04-bioc-3.18` to get the latest 3.18,
17-
regardless of R version, or even simply `jammy`/`22.04` to get the latest release container.
17+
`[ubuntu_version]-bioc-[bioc_version]-r-[r_version]` eg `noble-bioc-3.22-r-4.5.0` or `24.04-bioc-3.22-r-4.5.0, where
18+
each level is optional. For example, one could use tag `noble-bioc-3.22` or `24.04-bioc-3.22` to get the latest 3.22,
19+
regardless of R version, or even simply `noble`/`24.04` to get the latest release container.
1820
`devel-` will prefix all devel container tags, followed by the same schema described above.
1921

2022
All containers will use the R command if no command is specified. Below are some examples for running the container.
2123
```
2224
# Interactive R session
23-
docker run -it ghcr.io/bioconductor/bioconductor_salt:jammy
25+
docker run -it ghcr.io/bioconductor/bioconductor_salt:noble
2426
# is equivalent to
25-
docker run -it ghcr.io/bioconductor/bioconductor_salt:jammy R
27+
docker run -it ghcr.io/bioconductor/bioconductor_salt:noble R
2628
2729
# Bash shell
28-
docker run -it ghcr.io/bioconductor/bioconductor_salt:jammy bash
30+
docker run -it ghcr.io/bioconductor/bioconductor_salt:noble bash
2931
3032
# Rscript
31-
docker run -it ghcr.io/bioconductor/bioconductor_salt:jammy "Rscript --version"
33+
docker run -it ghcr.io/bioconductor/bioconductor_salt:noble "Rscript --version"
34+
```
35+
36+
We are also experimentally building BBS-like containers based on Nvidia
37+
containers, which can be run as
38+
3239
```
40+
docker run --gpus all -it ghcr.io/bioconductor/bioconductor_salt:devel-nvidia-noble R
41+
```
42+
43+
#### Note for containers with an Nvidia base
44+
45+
This software contains source code provided by NVIDIA Corporation.
46+
47+
These containers are subject to
48+
https://developer.download.nvidia.com/licenses/NVIDIA_Deep_Learning_Container_License.pdf.
49+
3350

3451
### Configuring for Ubuntu 22.04
3552

‎saltstack/pillar/common/init.sls‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,18 @@
11
{% from '../custom/init.sls' import branch, version, environment,
22
r_download, r_version, r_previous_version, cran_mirror, cycle, name,
3-
create_users, machine_type %}
3+
create_users, machine_type, gpu %}
44

55
{% if machine_type == 'standalone' %}
66
{# Assuming salt is run is /Users/a_user or /home/a_user, take the last
77
directory as the user name #}
8-
{% set build_user = grains['cwd'].split("/")[-1] %}
9-
{% else %}
8+
{% set build_user = grains['cwd'].split("/")[-1] or 'biocbuild' %}
9+
{% endif %}
10+
1011
{% if create_users %}
1112
{% from '../custom/init.sls' import biocbuild_password, biocbuild_key,
1213
biocbuild_authorized_key, biocpush_password, biocpush_key,
1314
biocpush_authorized_key %}
1415
{% endif %}
15-
{% set build_user = 'biocbuild' %}
16-
{% endif %}
1716

1817
{%- if branch == 'release' %}
1918
{% set current_branch = 'RELEASE_' ~ version.replace(".", "_") %}
@@ -42,6 +41,7 @@ build:
4241

4342
machine:
4443
name: {{ name }}
44+
gpu: {{ gpu }}
4545
env: {{ environment }}
4646
slash: {{ slash }}
4747
ip: 127.0.1.1

‎saltstack/pillar/custom/devel_standalone.sls‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@
1111
{% set name = 'bbs-machine' %}
1212
{% set create_users = False %}
1313
{% set machine_type = 'standalone' %}
14+
{% set gpu = False %}

‎saltstack/pillar/custom/devel_standalone_gpu.sls‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{# Custom Settings #}
2+
3+
{% set branch = 'devel' %} {# Use 'release' or 'devel' #}
4+
{% set version = '3.22' %}
5+
{% set environment = 'dev' %} {# Use 'dev' or 'prod' #}
6+
{% set r_download = 'https://cran.r-project.org/src/base/R-4/R-4.5.0.tar.gz' %}
7+
{% set r_version = 'R-4.5' %}
8+
{% set r_previous_version = 'R-4.4' %}
9+
{% set cran_mirror = 'https://cloud.r-project.org/' %}
10+
{% set cycle = 'devel' %} {# Use 'devel' for Spring to Fall, 'patch' for Fall to Spring #}
11+
{% set name = 'bbs-machine' %}
12+
{% set create_users = False %}
13+
{% set machine_type = 'standalone' %}
14+
{% set gpu = True %}

‎saltstack/pillar/custom/release_standalone.sls‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@
1111
{% set name = 'bbs-machine' %}
1212
{% set create_users = False %}
1313
{% set machine_type = 'standalone' %}
14+
{% set gpu = False %}

‎saltstack/pillar/custom/release_standalone_gpu.sls‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{# Custom Settings #}
2+
3+
{% set branch = 'release' %} {# Use 'release' or 'devel' #}
4+
{% set version = '3.21' %}
5+
{% set environment = 'dev' %} {# Use 'dev' or 'prod' #}
6+
{% set r_download = 'https://cran.r-project.org/src/base/R-4/R-4.5.0.tar.gz' %}
7+
{% set r_version = 'R-4.5' %}
8+
{% set r_previous_version = 'R-4.4' %}
9+
{% set cran_mirror = 'https://cloud.r-project.org/' %}
10+
{% set cycle = 'patch' %} {# Use 'devel' for Spring to Fall, 'patch' for Fall to Spring #}
11+
{% set name = 'bbs-machine' %}
12+
{% set create_users = False %}
13+
{% set machine_type = 'standalone' %}
14+
{% set gpu = True %}

‎saltstack/salt/common/linux.sls‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,12 @@ change_host:
1919
- clean: True
2020
{% endif %}
2121

22+
make_user_{{ machine.user.name }}:
23+
user.present:
24+
- name: {{ machine.user.name }}
25+
- home: {{ machine.user.home }}/{{ machine.user.name }}
26+
- shell: {{ machine.user.shell }}
27+
2228
{% if machine.create_users %}
2329
{% if machine.additional is defined %}
2430
{% set groups = machine.groups + machine.additional.groups %}
@@ -80,9 +86,26 @@ git_clone_{{ repo.bbs.name }}_to_{{ machine.user.home }}/{{ machine.user.name }}
8086
- target: {{ machine.user.home }}/{{ machine.user.name }}/{{ repo.bbs.name }}
8187
- user: {{ machine.user.name }}
8288

89+
{%- if machine.gpu %}
90+
{% for pkg_type in ["required_compile_R", "optional_compile_R", "cran", "bioc"] %}
91+
install_{{ pkg_type }}_pkgs:
92+
cmd.run:
93+
- name: DEBIAN_FRONTEND=noninteractive apt-get -y install $(cat /home/{{ machine.user.name }}/{{ repo.bbs.name }}/{{ grains["os"] }}-files/{{ grains["osrelease"] }}/apt_{{ pkg_type }}.txt | awk '/^[^#]/ {print $1}')
94+
{%- endfor %}
95+
96+
install_pkgs_for_gpu:
97+
pkg.installed:
98+
- pkgs:
99+
- pandoc
100+
- texlive-latex-base
101+
- texlive-fonts-extra
102+
- libthrust-dev
103+
- libcub-dev
104+
{% else %}
83105
install_apt_pkgs:
84106
cmd.run:
85107
- name: DEBIAN_FRONTEND=noninteractive apt-get -y install $(cat /home/{{ machine.user.name }}/{{ repo.bbs.name }}/{{ grains["os"] }}-files/{{ grains["osrelease"] }}/apt_*.txt | awk '/^[^#]/ {print $1}')
108+
{%- endif %}
86109

87110
check_locale:
88111
locale.present:

‎saltstack/salt/rlang/linux.sls‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,13 @@
55
{% set extracted_directory = r.download.split("/")[-1][:-7].split("_")[0] %}
66
{% set bbs_bioc = machine.user.home ~ "/" ~ machine.user.name ~ "/bbs-" ~ "%.2f" | format(build.version) ~ "-bioc" %}
77

8-
get_R_tarball:
9-
cmd.run:
10-
- name: curl -L {{ r.download }} -o {{ bbs_bioc }}/rdownloads/{{ tarball }}
11-
- creates: {{ bbs_bioc }}/rdownloads/{{ r.version }}.tar.gz
12-
13-
fix_permissions_ownership_R_tarball:
8+
get_R_tarball:
149
file.managed:
1510
- name: {{ bbs_bioc }}/rdownloads/{{ tarball }}
11+
- source: {{ r.download }}
1612
- user: {{ machine.user.name }}
1713
- group: {{ machine.user.name }}
14+
- skip_verify: True
1815

1916
R_tarball_extracted:
2017
archive.extracted:

0 commit comments

Comments
 (0)