Why do some environment variables not propagate to kernels (I.e. LD_LIBRARY_PATH)?

[ehvr20]

Hi all,

I’m running Kallichore inside a Singularity container and trying to use GPU/CUDA functionality via the cupy Python library. After a frustrating few hours of debugging, I discovered that $LD_LIBRARY_PATH—which the CUDA runtime relies on to locate the drivers—was not being propagated correctly to spawned kernels.

Specifically:

• $LD_LIBRARY_PATH inside the container differs from what kernels launched from the container see.
• I had to manually define it in .bashrc and enable Kernel Supervisor: Run in Shell to make things work.
• Interestingly, some environment variables, like $SINGULARITY_NAME, are propagated correctly, but $LD_LIBRARY_PATH is reset.

I’ve seen a few related issues but nothing directly addressing this, so I’m not opening a bug report straightaway.

Question: Why are certain environment variables propagated correctly while $LD_LIBRARY_PATH is not?

Thanks in advance for any insights!

[jmcphers]

Thanks for reporting! There are a lot of layers that have an opportunity to modify environment variables, so a little more detail might help:

• Can you share a screenshot of this happening?
• If you create a new Terminal in Positron, do you see the LD_LIBRARY_PATH you expect there (after removing your .bashrc workaround of course)?
• Do you see any mentions of LD_LIBRARY_PATH in the terminal environment contributions? (To see those, with your Terminal open run the command Terminal: Show Environment Contribtions)
• Where/how are you setting SINGULARITY_NAME?

Kallichore loads all terminal environment variables and so my initial guess is that one of the extensions is setting LD_LIBRARY_PATH in such a way that it overrides the value inside the container.

[ehvr20]

I suspect this might be a personal environment issue rather than something directly related to Positron. As you said, it’s complicated. I can’t provide a screenshot, but here are the outputs:

1. Interactive Console (Python) - note: this exists in a container via Kallichore

>> import os; os.environ['LD_LIBRARY_PATH']
'/usr/local/cuda-12.8/extras/CUPTI/libr64:/usr/local/cuda-12.8/lib64
>> os.environ['SINGULARITY_NAME']
'uv_r_cuda_0.1.sif'
>> import subprocess; subprocess.run('nvidia-smi')
NVIDIA_SMI couldn't find libnvidia-ml.so library in your system. # ... etc

2. Terminal

> singularity shell --nv MYIMAGE.sif # launch an interactive shell in my container
> echo $LD_LIBRARY_PATH
/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
> echo $SINGULARITY_NAME
uv_r_cuda_0.1.sif
> nvidia-smi
# Output is large so tldr - SUCCESS

Worth noting: terminals are not managed by the Kallichore server, so I have to launch a container manually with singularity shell.
The issue is that nvidia-smi fails (indicating the GPU is inaccessible) when /.singularity.d/libs drops out of $LD_LIBRARY_PATH. This only happens in kernels launched from Kallichore.

The $SINGULARITY_NAME variable is added when launching a singularity container and it is just a convenient way to confirm you’re in a Singularity container. It was an example of an environment variable that should propagate from the container to Kallichore and then onto the kernel sessions.

---

As I said before, I found a way to solve this by manually adding the /.singularity.d/libs to the $LD_LIBRARY_PATH to the .bashrc and forcing new kernels to be login shells to launch ~/.bashrc. Thereby fixing it - I just want to check if there was something obvious you were aware of.

---

I have, however, discovered a massive flaw with my Kallichore container strategy that I need some help with but I am hesitant to submit an issue or flood the discussion with questions when using positron in such an unintended way. I will ask it here but create an issue or new discussion if preferable.

The problem is that interpreter detection happens outside Kallichore/my container. My container provides an R interpreter at /usr/bin/R, but the remote host doesn’t have one. That means while Kallichore could run an R kernel, Positron can’t see or launch it.

I tried creating a “fake” R binary on the host that symlinked to /usr/bin/R inside the container initially, but the validity check failed. What I really need is a way to tell Positron, “there is an R binary at /usr/bin/R that will work via Kallichore,” even though it can’t see it directly.

So my question is: is there any way to manually add interpreters, or is this essentially the limit of what Positron can support in this setup? Or could my "fake" R symlink approach be improved, such that Positron sees it as valid?

Thanks for the help!

[jmcphers]

You're right, that's going to be challenging! We haven't built anything that will allow you to do this elegantly since we've not tried to support this kind of remote configuration. I think you're going to find it will be difficult to get this to work as you expect, especially as you'll also need e.g. the Ark kernel in your container at the right path. The LSP may also struggle to read code/paths outside your current one since it doesn't have filesystem access, etc.

This is all written up here: #8300 if you'd like to comment/upvote/share more about what you're trying to accomplish!

[ehvr20]

What a bummer! It was almost perfect :D

I have a follow up question regarding this - could I manually launch kernels? If I manually launch an IRkernel, would the positron kernel supervisor see it and be able to use it? (even if language servers etc won't work)

More generally when launching both python and R kernels manually can they be used in Positron similar to VSCode?

Thanks!