Error in running slurm web on localhost

[Talavig]

I try to run slurm-web on a single, localhost cluster as a test. when i try to go to localhost:5011 from my browser, i get the following message:
Server error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
when i look at the logs for slurm-web-agen, i see the following:
Thread-858 (process_request_thread): [INFO] 127.0.0.1 - - [14/Jul/2024 13:58:32] "GET /v3.1.0/stats HTTP/1.1" 500 - Jul 14 13:58:32 standalone slurm-web-agent[7637]: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Jul 14 13:58:32 standalone slurm-web-agent[7637]: return self.scan_once(s, idx=_w(s, idx).end()) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode Jul 14 13:58:32 standalone slurm-web-agent[7637]: obj, end = self.raw_decode(s) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode Jul 14 13:58:32 standalone slurm-web-agent[7637]: return _default_decoder.decode(s) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 525, in loads Jul 14 13:58:32 standalone slurm-web-agent[7637]: return complexjson.loads(self.text, **kwargs) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json Jul 14 13:58:32 standalone slurm-web-agent[7637]: result = response.json() Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 51, in slurmrest Jul 14 13:58:32 standalone slurm-web-agent[7637]: items = func(*args) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 76, in filter_fields Jul 14 13:58:32 standalone slurm-web-agent[7637]: return func(*args) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 88, in _cached_data Jul 14 13:58:32 standalone slurm-web-agent[7637]: return _cached_data( Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 101, in _cached_jobs Jul 14 13:58:32 standalone slurm-web-agent[7637]: for job in _cached_jobs(): Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 236, in stats Jul 14 13:58:32 standalone slurm-web-agent[7637]: return view(*args, **kwargs) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/rfl/web/tokens.py", line 93, in wrapped Jul 14 13:58:32 standalone slurm-web-agent[7637]: return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request Jul 14 13:58:32 standalone slurm-web-agent[7637]: rv = self.dispatch_request() Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request Jul 14 13:58:32 standalone slurm-web-agent[7637]: rv = self.handle_user_exception(e) Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request Jul 14 13:58:32 standalone slurm-web-agent[7637]: response = self.full_dispatch_request() Jul 14 13:58:32 standalone slurm-web-agent[7637]: File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
Its important to mention that i only manage to reach slurmrestd by using sudo in my curl command. maybe that's related?
Why is this error thrown?
In addition, can you only reach slurmrestd through a unix socket? is it not possible to open it up to a localhost http communication? because now, the way I access the slurm socket through the ui without running it with sudo is to chmod 777 slurmrestd.socket. before i did it, got an error saying Permission denied.
Thanks in advance:)

[rezib]

Hi @Talavig, I converted the issue into a Q&A discussion thread as it is arguably more a set of questions than an actual bug.

Its important to mention that i only manage to reach slurmrestd by using sudo in my curl command. maybe that's related?
Why is this error thrown?

The error your are facing could be explained by permission issue on slurmrestd Unix socket. In the documented configuration, we run both slurmrestd and Slurm-web agent with slurm system user. This has to be defined with a drop-in configuration override for slurmrestd systemd service, this is configured by default in Slurm-web agent service. This way, Slurm-web agent has the permission to access slurmrestd Unix socket.

In addition, can you only reach slurmrestd through a unix socket? is it not possible to open it up to a localhost http communication?

As stated in architecture documentation, Slurm-web only supports the Unix socket for the moment. It might change in the future with #313.

[Talavig]

When I try to run slurmrestd as user slurm, I get the following error:

Jul 16 11:10:27 standalone systemd[1]: slurmrestd.service: Failed with result 'exit-code'.
Jul 16 11:10:27 standalone systemd[1]: slurmrestd.service: Main process exited, code=exited, status=1/F>
Jul 16 11:10:27 standalone slurmrestd[2052]: slurmrestd: fatal: slurmrestd should not be run as SlurmUs>
Jul 16 11:10:27 standalone slurmrestd[2052]: fatal: slurmrestd should not be run as SlurmUser
Jul 16 11:10:27 standalone slurmrestd[2052]: debug:  main: server listen mode activated
Jul 16 11:10:27 standalone slurmrestd[2052]: debug:  NodeNames=localhost setting Sockets=4 based on CPU>
Jul 16 11:10:27 standalone slurmrestd[2052]: debug:  Reading slurm.conf file: /etc/slurm/slurm.conf
Jul 16 11:10:27 standalone slurmrestd[2052]: debug:  slurm_conf_init: using config_file=/etc/slurm/slur>
Jul 16 11:10:27 standalone slurmrestd[2052]: debug2: _establish_config_source: using config_file=/etc/s>
Jul 16 11:10:27 standalone systemd[1]: Started Slurm REST daemon.

This goes directly against what you have told me to do.
I don't know why this happens. Is there a workaround for this? maybe give the slurm-web user permissions to use the socket?
The following is my unit file for slurmrestd:

[Unit]
Description=Slurm REST daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurm.conf
[Service]
ExecStart=
Environment=
ExecStart=/usr/sbin/slurmrestd -vvv unix:/var/run/slurmrestd/slurmrestd.socket
RuntimeDirectory=slurmrestd
RuntimeDirectoryMode=0755
User=slurm
Group=slurm

[rezib]

What version of Slurm are you running?

Can you show me the output of these commands:

• systemctl cat slurmrestd.service
• scontrol show config | grep SlurmUser
• getent passwd slurm

Can you try to add Environment=SLURMRESTD_SECURITY=disable_user_check in your slurmrestd service override?

I have never met the error fatal: slurmrestd should not be run as SlurmUser in my testing environments, I would like to understand why.

[Talavig]

I have checked what version we run using sinfo -V, we are running slurm-wlm 21.08.5
here are all the commands you have asked for:
$ systemctl cat slurmrestd.service

# /lib/systemd/system/slurmrestd.service
[Unit]
Description=Slurm REST daemon
After=network-online.target munge.service slurmctld.service
Wants=network-online.target
ConditionPathExists=/etc/slurm/slurm.conf
Documentation=man:slurmrestd(8)

[Service]
Type=simple
EnvironmentFile=-/etc/default/slurmrestd
# Default to local auth via socket
ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS unix:/run/slurmrestd.socket
# Uncomment to enable listening mode
#Environment="SLURM_JWT=daemon"
#ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS 0.0.0.0:6820
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/slurmrestd.service.d/override.conf
[Unit]
Description=Slurm REST daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurm.conf


[Service]
#User=slurmrestuser
#Group=slurmrestgroup
#ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS
#Environment=SLURMRESTD_LISTEN=:6820

ExecStart=
Environment=
ExecStart=/usr/sbin/slurmrestd -vvv unix:/var/run/slurmrestd/slurmrestd.socket
Environment=SLURMRESTD_SECURITY=disable_user_check
#ExecStart=/usr/sbin/slurmrestd -vvv localhost:8080 unix:/var/run/slurmrestd/slurmrestd.socket
#ExecStart=/usr/sbin/slurmrestd -vvv unix:/var/run/slurmrestd/slurmrestd.socket
#Environment=SLURMRESTD_LISTEN=localhost:6820,unix:/var/run/slurmrestd/slurmrestd.socket
#Environment=SLURMRESTD_LISTEN=:6820
#Environment=SLURM_JWT=daemon
RuntimeDirectory=slurmrestd
RuntimeDirectoryMode=0755
User=slurm
Group=slurm

$ scontrol show config | grep SlurmUser

SlurmUser               = slurm(64030)

getent passwd slurm

slurm:x:64030:64030::/nonexistent:/usr/sbin/nologin

In addition, I tried adding the env var you told me in the override, and I get the following error in the agent:

Thread-38 (process_request_thread): [INFO] 127.0.0.1 - - [16/Jul/2024 13:28:26] "GET /v3.1.0/stats HTTP/1.1" 500 -
Jul 16 13:28:26 standalone slurm-web-agent[2310]: KeyError: 'warnings'
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     if len(result["warnings"]):
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 64, in slurmrest
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     items = func(*args)
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 76, in filter_fields
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     return func(*args)
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 88, in _cached_data
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     return _cached_data(
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 101, in _cached_jobs
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     for job in _cached_jobs():
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 236, in stats
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     return view(*args, **kwargs)
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/rfl/web/tokens.py", line 93, in wrapped
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     rv = self.dispatch_request()
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     rv = self.handle_user_exception(e)
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
Jul 16 13:28:26 standalone slurm-web-agent[2310]:     response = self.full_dispatch_request()
Jul 16 13:28:26 standalone slurm-web-agent[2310]:   File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
Jul 16 13:28:26 standalone slurm-web-agent[2310]: Traceback (most recent call last):
Jul 16 13:28:26 standalone slurm-web-agent[2310]: Thread-38 (process_request_thread): [ERROR] Exception on /v3.1.0/stats [GET]

[rezib]

I have checked what version we run using sinfo -V, we are running slurm-wlm 21.08.5 here are all the commands you have asked

OK, unfortunately you are actually running a version of Slurm that is too old… As stated in the quickstart guide, Slurm-web requires Slurm >= 23.02.

BTW, I suspect a regression in Slurm >= 22.05.7 introduced with commit SchedMD/slurm@0a07c53 that may cause slurm user fatal error to never be triggered in more recent versions of Slurm. I will check this in more depth.

I will also try to do something in Slurm-web to report clearer error when the version of Slurm is not supported.

[rezib]

For the record, I have just opened #316, #319 and #320 following this discussion.

[Talavig]

After many agonizing hours, I finally managed to upgrade slurm (I'm new to slurm).
When running sinfo -V, I get that my version now is 23.11.8, which should work according to what you have told me.
I now try to access the ui and i get the following error:

Thread-196 (process_request_thread): [INFO] 127.0.0.1 - - [12/Aug/2024 13:18:53] "GET /v3.1.0/stats HTTP/1.1" 500 -
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    return self.scan_once(s, idx=_w(s, idx).end())
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 525, in loads
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
    result = response.json()
  File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 51, in slurmrest
    items = func(*args)
  File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 76, in filter_fields
    return func(*args)
  File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 88, in _cached_data
    return _cached_data(
  File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 101, in _cached_jobs
    for job in _cached_jobs():
  File "/usr/lib/python3/dist-packages/slurmweb/views/agent.py", line 236, in stats
    return view(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/rfl/web/tokens.py", line 93, in wrapped
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
    response = self.full_dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app

I tried running the commands you supplied to me earlier in our discussion, nothing seems to have been changed.
Do you have any idea why this may happen?

[rezib]

@Talavig can you give me the output of these commands:

• systemctl cat slurmrestd.service
• curl --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.39/jobs

[Talavig]

systemctl cat slurmrestd.service

# /lib/systemd/system/slurmrestd.service
[Unit]
Description=Slurm REST daemon
After=network-online.target munge.service slurmctld.service
Wants=network-online.target
ConditionPathExists=/etc/slurm/slurm.conf
Documentation=man:slurmrestd(8)

[Service]
Type=simple
EnvironmentFile=-/etc/default/slurmrestd
# Default to local auth via socket
ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS unix:/run/slurmrestd.socket
# Uncomment to enable listening mode
#Environment="SLURM_JWT=daemon"
#ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS 0.0.0.0:6820
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/slurmrestd.service.d/override.conf
[Unit]
Description=Slurm REST daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurm.conf


[Service]
#User=slurmrestuser
#Group=slurmrestgroup
#ExecStart=/usr/sbin/slurmrestd $SLURMRESTD_OPTIONS
#Environment=SLURMRESTD_LISTEN=:6820

ExecStart=
Environment=
ExecStart=/usr/sbin/slurmrestd -vvv unix:/var/run/slurmrestd/slurmrestd.socket
Environment=SLURMRESTD_SECURITY=disable_user_check
#ExecStart=/usr/sbin/slurmrestd -vvv localhost:8080 unix:/var/run/slurmrestd/slurmrestd.socket
#ExecStart=/usr/sbin/slurmrestd -vvv unix:/var/run/slurmrestd/slurmrestd.socket
#Environment=SLURMRESTD_LISTEN=localhost:6820,unix:/var/run/slurmrestd/slurmrestd.socket
#Environment=SLURMRESTD_LISTEN=:6820
#Environment=SLURM_JWT=daemon
RuntimeDirectory=slurmrestd
RuntimeDirectoryMode=0755
User=slurm
Group=slurm

curl --unix-socket /run/slurmrestd/slurmrestd.socket http://slurm/slurm/v0.0.39/jobs

Authentication failure

I would like to mention that when running the curl command using the slurm user or simply sudo it works properly, and slurm agent runs with the slurm user (checked using systemctl cat slurm-web-agent.service).