Singularity/apptainer

[joergfunger]

In the merge by @jpthiele and the current pull request by @Sarbani-Roy , apptainer is added in the github actions as an additional tool to be installed. @srosenbu pointed out that this could be also added to the conda environment, where snakemake etc are installed, just by adding apptainer there. This would be IMO more straightforward, because in a real test case on our machine, not every user has to install apptainer on the system level, but we just provide that in the conda environment (which we have to build anyway). What do you think?

[jpthiele]

While I prefer not to use conda on a system level I think it makes sense to add it to a general "benchmarkplatform" environment.
Or at least try it and see if there are side-effects. If there are none we should include it.
For those happy to use conda for these tools, it gets straightforward. For those who don't, all requirements are recorded in one place.

[srosenbu]

I would volunteer to try to include apptainer into the benchmark environment. Should only be a few minutes. We can discuss then in the PR

[srosenbu]

So, this does not seem to be straightforward. With apptainer being installed via conda, I get errors that I can't explain. Feel free to look into the workflows in the PR. It seems to me that one issue is that we need to have fuse2fs installed which is not available on conda-forge.

My main reason for pushing this idea was mainly that if everything runs through conda, no one would need sudo rights, just a linux system and the compute environment could be setup with a singe definition file. But if we now have to install fuse2fs via apt-get, we are back at the same problem.

[joergfunger]

from the apptainer docu it says "allow setuid-mount extfs permits/denies execution of ext3 filesystems in SUID mode using the kernel ext4 driver, both inside and outside of SIF files. When set to no, fuse2fs will be used instead. For security reasons this defaults to no." Can we set that to yes?

[srosenbu]

I am not sure. From the docs it seems like this requires some admin rights. Someone in this issue suggests to install apptainer system wide for use with snakemake snakemake/snakemake#304.

The good thing is that we can still run it without sudo, but someone might need to install aptainer in sudo mode.

[joergfunger]

But then we are back at the question of admin rights, so can I only run apptainer when (at least for some depending tools) an admin installs the packages (I thhought this is exactly the advantage of apptainer that we do not need admin rights). I see the advantage that we can instantiate a container without admin rights (which would not be possible with docker, I guess), but mmh - feels subotimal to me.

[srosenbu]

I agree, i'd rather have everything run inside conda. Although, having someone install Apptainer once and then never needing sudo-rights again to run a container is still a benefit.

I might investigate apptainer more this month since I want to try our HPC-Cluster.