Is libnss_wrapper required for custom images?

[asmacdo]

We are using custom jupyterhub images that does not have libnss-wrapper installed.

When I launch a user pod and open a terminal in jupyterhub, I see (maybe 20x):

ERROR: ld.so: object 'libnss_wrapper.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored. 

This requirement seems to be hardcoded in the profiles

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py

Line 347 in 86d5d08

"LD_PRELOAD": "libnss_wrapper.so",

Questions:

• Should we modify our custom images to install libnss_wrapper?
• Is there any documentation on other nebari-specific image requirements?
• alternatively is it safe to override that environment variable (see below)

singleuser:
  extraEnv:
    LD_PRELOAD: ""

[marcelovilla]

Hey @asmacdo,

I am not aware of what the exact purpose of libnss_wrapper in JupyterLab is but maybe @krassowski could shed some light here. We're definitely installing it in our Docker images: https://github.com/nebari-dev/nebari-docker-images/blob/d7f55a1e34d582cc1711d0dc701a2e4d5e3fe008/Dockerfile#L78

Regarding your questions:

Should we modify our custom images to install libnss_wrapper?

Unless there's a good reason not to do that, I don't see why that would be a problem.

Is there any documentation on other nebari-specific image requirements?

I don't think we have anything specific on our docs about requirements for the images, but you can take a look at the nebari-docker-images repo to see what things are being installed and how they're being configured. If you need to build custom images for your deployment, I'd suggest building them on top of these ones as they are the ones we test and support.

alternatively is it safe to override that environment variable (see below)

My guess is we're using libnss_wrapper to emulate username and user ID resolution within the containers running the JupyterLab servers. If that's the case, I wouldn't be surprised if you ran into permission issues. I haven't tested it myself so I cannot say for certain.

[viniciusdc]

@marcelovilla @asmacdo The use of libnss_wrapper in the JupyterLab configuration is essential for managing user and group IDs (UID/GID) within the containerized environment. It’s configured through nss_wrapper as part of the spawner setup in Nebari:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/files/jupyterhub/03-profiles.py

Lines 343 to 364 in ee1db3e

	def configure_user(username, groups, uid=1000, gid=100):
	environment = {
	# nss_wrapper
	# https://cwrap.org/nss_wrapper.html
	"LD_PRELOAD": "libnss_wrapper.so",
	"NSS_WRAPPER_PASSWD": "/tmp/passwd",
	"NSS_WRAPPER_GROUP": "/tmp/group",
	# default files created will have 775 permissions
	"NB_UMASK": "0002",
	# set default shell to bash
	"SHELL": "/bin/bash",
	# set home directory to username
	"HOME": f"/home/{username}",
	# Disable global usage of pip
	"PIP_REQUIRE_VIRTUALENV": "true",
	}
	etc_passwd, etc_group = generate_nss_files(
	users=[{"username": username, "uid": uid, "gid": gid}],
	groups=[{"groupname": "users", "gid": gid}],
	)

Since Nebari provisions users dynamically, managed via Keycloak, we need a way to define those users within the JupyterLab container so that file permissions, group-based access (e.g., shared folders), and kernel interactions behave correctly. Rather than modifying the container image or creating system users manually, nss_wrapper allows us to redirect system calls that resolve user and group information (e.g., getpwuid, getgrgid) to temporary in-memory files (/tmp/passwd, /tmp/group) at runtime.

This mechanism has been part of our ecosystem for over five years. Disabling it may lead to unexpected behavior in permission handling, Conda environment resolution, or kernel spawning.

While plans have been made to migrate some of these responsibilities from the container image into application logic using tools like skel files, nss_wrapper remains a critical component of the current architecture and is strongly recommended in any image used within Nebari.

There is another open issue that might touch on this requirement in the future
#3044
Also, initially we relied on the jovyan user instead of the dynamically created ones, so this also might be useful to check out for history context
#1138