Running quarto in singularity

[laurapoggio-sptools]

I use singularity containers for many projects. I tested quarto with python from within one such container with the python example from here

singularity exec  ./my_image quarto render tests_quarto_py.qmd --to html

I get the following error:

ERROR: Read-only file system (os error 30), mkdir '/run/user/1000/jt'

Jupyter notebook works fine from the same singularity container and the command works fine if using docker. The issue seems to be that the file system of Singularity is read-only (if not in specific locations).

Is there a way to tell quarto not to use the standard /run/user/1000/jt? An option to specify the folder to somewhere writable?
Thanks

[mcanouil]

Would you have a working reproducible example?
/run/user/1000 is a temporary directory created to run the job, you should not have an issue.
Are you using any particular setup in Singularity? Disk mounts or else?

Main default difference between Docker and Singularity is that Docker is using root privileges by default while Singularity is using root-less mode by default.
Maybe this is somehow the issue.

[nitin-ppnp]

Hi, I am facing the same issue. Is it possible to manually set a path for the temporary files?

My use-case: I have my code in my home folder, while all the dependencies are set up in a singularity container. I start the container and the home directory is mounted by default. In this case, I want to write documentation for my code/website which include code snippets too. I find quarto to be an attractive solution. I have quarto repo cloned and built in my home directory. I can use the built binary from within the container, and it works fine if there is no code in the .qmd file. However, if I add a code block, I face the same error as mentioned above. I suspect when run from the container, quarto tries to make a directory in the /run which is mounted read-only.

[mcanouil]

@nitin-ppnp Did you try to setup your container accounting for this as described in the links I provided?
#2559 (reply in thread)

[nitin-ppnp]

Sorry, I tried those options, nothing is working but running the container as root seems to be working. I am facing another issue related to jupyter kernel now.
Starting python3 kernel...WARNING: Unable to start Jupyter kernel for /home/nitin/projects/measurements_to_bodies_w_star/measureme/docs/index.ipynb
The process end after this warning and nothing happen afterwards.

[mcanouil]

I think it would be best to open a new Q&A discussion with a complete description of the issue.
This means, operating system, Quarto version, environment, and a reproducible example or steps to reproduce your error.
Without those information, it's going to be hard or impossible to help.

[laurapoggio-sptools]

the call to singularity was
singularity exec ./my_image quarto render tests_quarto_py.qmd --to html

My image was build as
singularity build --sandbox singularity_image docker://docker_image
I am aware that sandbox can sometimes create problems but it is the only way that I can use singularity on the HPC I have access to

I found a work-around the issue with a slightly different call:
singularity exec --writable ./my_image quarto render tests_quarto_py.qmd --to html
Maybe it would be good to have an option to specify where the temporary directory to run the job can go?

I will try to upload a (small) reproducible image to docker hub as soon as I can.

test_quarto_py.qmd is

---
title: "Quarto Basics"
format:
  html:
    code-fold: true
jupyter: python3
---

For a demonstration of a line plot on a polar axis, see @fig-polar.

```{python}
#| label: fig-polar
#| fig-cap: "A line plot on a polar axis"

import numpy as np
import matplotlib.pyplot as plt

r = np.arange(0, 2, 0.01)
theta = 2 * np.pi * r
fig, ax = plt.subplots(
  subplot_kw = {'projection': 'polar'} 
)
ax.plot(theta, r)
ax.set_rticks([0.5, 1, 1.5, 2])
ax.grid(True)
plt.show()

[mcanouil]

Just to be sure, are you using any Singularity user/global config files in addition?
Also, you might want to control what bind mounts you are using, according to the admin settings, this can be the issue (see https://docs.sylabs.io/guides/latest/user-guide/bind_paths_and_mounts.html#disabling-system-binds), i.e., there might be some user home automatic bind ongoing, which if the home does not exist leads to error.

You might want to checkout the --no-home and --containall flags (see https://docs.sylabs.io/guides/latest/user-guide/bind_paths_and_mounts.html#using-no-home-and-containall-flags).

[laurapoggio-sptools]

Thanks a lot for the tips. I am checking out if these completely solve the issue.

[laurapoggio-sptools]

I finally managed to do some tests. The user $HOME is mounted by default, but there are no other system mounts or binds.

singularity exec --no-home gives an error: error: Could not create TypeScript compiler cache location: "/home/user/.cache/deno/gen" and the image does not even start

singularity exec --containall can not find tests_quarto_py.qmd when it is not in the image itself but sourced from a local mount. It works if the qmd is in the image itself, but it is not a very common case for us.

[mcanouil]

I am not Singularity expert, I only used it once.
Unfortunately, I do not have right now any setup with it to make some tests.

I still think, the issue here can be resolved by identifying/tweaking Singularity setup when executing/deploying container.

[stitam]

Hi, I am wondering if this has been resolved? I have a similar issue: I must run R stuff using the --no-home argument for reasons discussed elsewhere (https://stackoverflow.com/questions/71252123/singularity-container-with-stringr-fails-only-locally-with-libicui18n-so-66-ca/75874025#75874025). When I run the singularity container with --no-home and try quarto --help I get this error: error: Could not create TypeScript compiler cache location: "/home/<username>/.cache/deno/gen" Check the permission of the directory. Understandable, I explicitly told singularity not to mount my /home. When I run it without the --no-home argument, it works, so for sure, the issue is about mounting a host directory for some quarto files. I did find a workaround to use my working directory for this: singularity run --no-home --bind .:/home/<username>/.cache container.img but this is not ideal because each user would have to use their own username so the command would always be different. Is it possible to tell quarto to use e.g. the working directory by default for generating/storing any quarto files? If not, would it be a reasonable feature request? Many thanks!

EDIT: Tools that generate files outside the working directory can be challenging to set up for singularity so if this could be avoided it would help singularity users a lot.

[chrstphdm]

Hi,

I was able to make it running by binding the folder :

singularity exec -B /tmp:/run/user/1000/jt ./my_image quarto render tests_quarto_py.qmd --to html

Hope it'll help ;-)

[grst]

If you are running into errors such as Could not create TypeScript compiler cache location: /home/sturm/.cache/deno/gen or ERROR: Read-only file system (os error 30), mkdir '/home/sturm/.local/share/quarto/logs, you can set XDG_..._HOME environment variables before running quarto, e.g.

export XDG_CACHE_HOME=/tmp/quarto_cache_home
export XDG_DATA_HOME=/tmp/quarto_data_home
quarto render report.qmd 

Note: A writeable directory is usually mounted into /tmp in a singularity container. You can of course also set the XDG variables to any other writable path in the container.

[medmaca]

Building on the above answers in order to get quarto to work in Apptainer/Singularity (pulled from a docker image) I found the following additions to the Dockerfile were required:

To deal with errors such as /opt/conda/bin/quarto: line 176: /root/miniforge3/bin/deno: No such file or directory:

# Fix Quarto for apptainer
ENV QUARTO_DENO /opt/conda/bin/deno
ENV QUARTO_DENO_DOM /opt/conda/lib/deno_dom.so
ENV QUARTO_PANDOC /opt/conda/bin/pandoc
ENV QUARTO_ESBUILD /opt/conda/bin/esbuild
ENV QUARTO_DART_SASS /opt/conda/bin/sass
ENV QUARTO_SHARE_PATH /opt/conda/share/quarto
ENV QUARTO_CONDA_PREFIX /opt/conda

To deal with errors such as:

Could not initialize cache database '/root/.cache/deno/node_analysis_cache_v1', deleting and retrying... (unable to open database file: /root/.cache/deno/node_analysis_cache_v1

Caused by:
    Error code 14: Unable to open the database file)
Failed to open cache file '/root/.cache/deno/node_analysis_cache_v1', opening in-memory cache.
Could not initialize cache database '/root/.cache/deno/dep_analysis_cache_v1', deleting and retrying... (unable to open database file: /root/.cache/deno/dep_analysis_cache_v1

Add

# Set relevant Cross Desktop Group settings
ENV XDG_RUNTIME_DIR /tmp
ENV XDG_CACHE_HOME=/tmp/quarto_cache_home
ENV XDG_DATA_HOME=/tmp/quarto_data_home

[mcanouil]

Thanks for sharing!

If you encounter /opt/conda/bin/quarto: line 176: /root/miniforge3/bin/deno: No such file or directory it means QUARTO_DENO has been set to use the system version of Deno which is unsafe.
If you change the PATH to Deno (by default QUARTO_DENO is unset because Quarto embeds Deno).
If you don't ensure you are using the same version of Deno that Quarto embeds, then you might completely break Quarto (https://github.com/quarto-dev/quarto-cli/blob/main/configuration).
Same applies with Pandoc, etc.

If you decide to use QUARTO_ environment variable, then this is where Quarto supports ends as the team cannot guarantee Quarto will work properly.

[medmaca]

Thanks for the quick reply.

Within a docker container this isn't an issue, but when using a apptainer container it doesn't carry out the usual activation of the conda environment and the ./conda/etc/conda/activate.d/quarto.sh file is never run, thus the QUARTO_* variables aren't set correctly which results in a /opt/conda/bin/quarto: line 176: /opt/conda/bin/tools/x86_64/deno: No such file or directory error. By manually setting the variables contained in ./conda/etc/conda/activate.d/quarto.sh within the Dockerfile it seems to circumvent this issue, and I believe should be using the correct version of deno.

[mcanouil]

That's a Conda issue/apptainer issue.

Note that the Conda recipe is not maintained by Quarto team.

• Are conda versions automatically updated? #5328

Did you try to manually activate the conda environment before running the Quarto CLI command? conda activate myenv.

There are alternatives to Conda such as using Quarto assets directly.

[medmaca]

Yes, conda and apptainer don't play well together.

At the moment quarto is part of a much larger container that uses conda, I will ultimately probably split quarto into a smaller separate container that doesn't use conda, but for now at least I've a working solution.

Thanks again for your quick responses.