Singularity gives "Unable to access jarfile error" where Docker runs successfully

[bskubi]

Version of Singularity
singularity-ce version 4.0.3-jammy

Describe the bug
I have a Dockerfile built and uploaded to my Docker Hub account at bskubi/juicer_tools:2.20.00

FROM openjdk:17
ADD https://github.com/aidenlab/Juicebox/releases/download/v2.20.00/juicer_tools.2.20.00.jar /app/juicer_tools.jar
RUN chmod +x /app/juicer_tools.jar
WORKDIR /app

Running java -jar juicer_tools.jar pre works fine with Docker but gives an "Unable to access jarfile" error with singularity.

To Reproduce
Either build the above Dockerfile with docker build -t juicer_tools ., tag and push it, and replace the docker image names as appropriate, or simply run the following:

docker run -it bskubi/juicer_tools:2.20.00 java -jar juicer_tools.jar pre --version | head -n 3
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARN [2024-01-19T09:26:44,986]  [Globals.java:138] [main]  Development mode is enabled
Juicer tools version 2.20.00

singularity pull docker://bskubi/juicer_tools:2.20.00
singularity exec juicer_tools_2.20.00.sif java -verbose -jar juicer_tools.jar pre --version | head -n 3
Error: Unable to access jarfile juicer_tools.jar

Expected behavior
The singularity exec command should give ~the same output as the docker run command.

OS / Linux Distribution

Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

This also occurs on our lab's HPC environment using a different singularity install singularity-ce version 3.10.0.

Installation Method
Ubuntu 22.04 (jammy) DEB package with dpkg

Other notes
Singularity has worked fine for converting and running several other docker images I've created recently. The main difference I can see is that this is the only one that requires running a java jarfile.

Also, running with sudo gives the error:

sudo singularity exec juicer_tools_2.20.00.sif java -jar /app/juicer_tools.jar pre --version
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARN [2024-01-19T09:40:48,225]  [Globals.java:138] [main]  Development mode is enabled
Exception in thread "main" java.lang.UnsatisfiedLinkError: /usr/java/openjdk-17/lib/libawt_xawt.so: libXext.so.6: cannot open shared object file: No such file or directory
	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
	at java.base/java.lang.Runtime.load0(Runtime.java:755)
	at java.base/java.lang.System.load(System.java:1953)
	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
	at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:315)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:285)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2398)
	at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
	at java.base/java.lang.System.loadLibrary(System.java:1989)
	at java.desktop/java.awt.Toolkit$2.run(Toolkit.java:1388)
	at java.desktop/java.awt.Toolkit$2.run(Toolkit.java:1386)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at java.desktop/java.awt.Toolkit.loadLibraries(Toolkit.java:1385)
	at java.desktop/java.awt.Toolkit.initStatic(Toolkit.java:1423)
	at java.desktop/java.awt.Toolkit.<clinit>(Toolkit.java:1397)
	at java.desktop/java.awt.Color.<clinit>(Color.java:277)
	at juicebox.HiCGlobals.<clinit>(HiCGlobals.java:43)
	at juicebox.tools.HiCTools.main(HiCTools.java:76)

[dtrudg]

Hello. This issue has been converted to a Q&A discussion as it is not really a bug, but is related to usage of Singularity.

Expected behavior
The singularity exec command should give ~the same output as the docker run command.

I'm afraid that, by design and due to the origins of Singularity and requirements of common workflows in HPC, this is not the expected behaviour for all containers run in Singularity's native mode.

As documented (https://docs.sylabs.io/guides/4.0/user-guide/singularity_and_docker.html#differences-and-limitations-vs-docker) Singularity does not exactly follow the behaviour of Docker, so issues should be expected with some containers.

The documentation section linked above includes details of the differences, potential workarounds, and also mentions the ne OCI-Mode (from Singularity 4.0) that is intended to more closely match the behaviour of Docker.

In this case, Singularity will not enter the container at the WORKDIR in the Dockerfile. Instead Singularity enters the container, by default, in the current working directory you are in on the host.

Add --cwd /app to the command line to enter the container at /app. Alternatively use the full path /app/juicer_tools.jar in your command.

[dtrudg]

NB - I'll make a note to specfically cover WORKDIR vs Singularity's CWD handling in the docs, given that it's not explicit there, so may not be clear.

[bskubi]

Thanks for your rapid response. Unfortunately the same issue reoccurs:

singularity exec juicer_tools_2.20.00.sif java -jar /app/juicer_tools.jar pre --version
Error: Unable to access jarfile /app/juicer_tools.jar

[bskubi]

OK, it looks like --oci mode works!

[dtrudg]

Glad to hear that --oci mode works.

Without --oci, there is another issue in play here too. If you want to pursue that... what do you see if you run...

singularity exec juicer_tools_2.20.00.sif ls -lah /app

Is the jar file visible, and does it have permissions that allow it to be run by your user account?